Jul 20 19:20:15.083998 ip-10-0-138-227 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 20 19:20:15.084012 ip-10-0-138-227 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 20 19:20:15.084023 ip-10-0-138-227 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 20 19:20:15.084360 ip-10-0-138-227 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 20 19:20:26.655328 ip-10-0-138-227 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 20 19:20:26.655345 ip-10-0-138-227 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1775f9895b804573aae0924fb09d5f7c --
Jul 20 19:22:26.709531 ip-10-0-138-227 systemd[1]: Starting Kubernetes Kubelet...
Jul 20 19:22:28.225677 ip-10-0-138-227 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 19:22:28.225677 ip-10-0-138-227 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 20 19:22:28.225677 ip-10-0-138-227 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 19:22:28.225677 ip-10-0-138-227 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 20 19:22:28.225677 ip-10-0-138-227 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 20 19:22:28.226313 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.225744    2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 20 19:22:28.232227 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232208    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 19:22:28.232227 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232227    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232233    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232237    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232240    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232243    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232246    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232249    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232252    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232255    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232258    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232261    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232263    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232266    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232269    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232272    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232275    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232278    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232281    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232284    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232287    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 19:22:28.232296 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232290    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232293    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232296    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232298    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232302    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232306    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232309    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232312    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232315    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232318    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232321    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232323    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232326    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232348    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232354    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232358    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232361    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232364    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232367    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232372    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 19:22:28.232818 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232375    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232379    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232382    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232386    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232389    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232392    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232395    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232398    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232401    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232404    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232407    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232410    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232414    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232417    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232420    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232423    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232426    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232429    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232431    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 19:22:28.233384 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232434    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232437    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232441    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232444    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232446    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232449    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232452    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232454    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232457    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232460    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232462    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232465    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232468    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232471    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232474    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232476    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232479    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232482    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232485    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 19:22:28.233853 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232489    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232492    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232496    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232499    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232502    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232505    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232509    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232512    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232515    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232518    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232521    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232524    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232527    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232530    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232532    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232535    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232538    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232541    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232544    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 19:22:28.234323 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.232548    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233291    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233298    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233301    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233304    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233309    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233313    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233316    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233319    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233322    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233325    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233347    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233354    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233359    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233362    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233365    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233368    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233371    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233374    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 19:22:28.234811 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233378    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233381    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233385    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233389    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233392    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233395    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233397    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233401    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233403    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233406    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233409    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233413    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233416    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233418    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233421    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233424    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233427    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233429    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233432    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 19:22:28.235297 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233436    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233438    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233441    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233444    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233447    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233450    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233452    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233456    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233458    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233461    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233464    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233467    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233469    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233473    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233475    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233479    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233482    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233485    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233487    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233490    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 19:22:28.235798 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233493    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233495    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233498    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233501    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233504    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233507    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233510    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233512    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233515    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233518    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233520    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233523    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233526    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233529    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233532    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233535    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233539    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233542    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233545    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233548    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 19:22:28.236314 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233551    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233553    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233556    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233559    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233562    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233565    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233568    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233570    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233573    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233576    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233578    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233581    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233584    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233586    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233589    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233592    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233596    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233599    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233602    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 19:22:28.236864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233606    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233609    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.233612    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234012    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234017    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234021    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234024    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234027    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234029    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234032    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234035    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234038    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234041    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234044    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234047    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234050    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234053    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234057    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234061    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234064    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 19:22:28.237359 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234067    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234070    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234073    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234076    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234078    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234081    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234084    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234087    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234090    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234094    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234098    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234101    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234105    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234108    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234112    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234114    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234122    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234126    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 19:22:28.237868 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234130    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234133    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234136    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234139    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234142    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234145    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234148    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234150    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234153    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234156    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234159    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234162    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234165    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234167    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234170    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234173    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234175    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234178    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234181    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234183    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 19:22:28.238382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234186    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234189    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234191    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234194    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234198    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234201    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234204    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234207    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234210    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234213    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234217    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234220    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234223    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234226    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234229    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234231    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234234    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234237    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234240    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234242    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 19:22:28.238887 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234245    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234248    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234251    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234254    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234257    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234259    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234262    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234265    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234268    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234270    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234273    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234276    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234279    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234281    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234284    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234287    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234290    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234292    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234295    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234298    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 19:22:28.239419 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234301    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234304    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234308    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.234311    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234410    2550 flags.go:64] FLAG: --address="0.0.0.0"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234419    2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234426    2550 flags.go:64] FLAG: --anonymous-auth="true"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234432    2550 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234436    2550 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234439    2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234444    2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234448    2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234452    2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234455    2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234459    2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234463    2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234466    2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234469    2550 flags.go:64] FLAG: --cgroup-root=""
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234473    2550 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234476    2550 flags.go:64] FLAG: --client-ca-file=""
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234479    2550 flags.go:64] FLAG: --cloud-provider="external"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234482    2550 flags.go:64] FLAG: --cluster-dns="[]"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234486    2550 flags.go:64] FLAG: --cluster-domain=""
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234489    2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 20 19:22:28.239979 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234493    2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234496    2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234500    2550 flags.go:64] FLAG: --container-log-max-files="5"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234504    2550 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234507    2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234510    2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234514    2550 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234518    2550 flags.go:64] FLAG: --contention-profiling="false"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234521    2550 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234524    2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234530    2550 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234533    2550 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234537    2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234540    2550 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234544    2550 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234547    2550 flags.go:64] FLAG: --enable-load-reader="false"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234550    2550 flags.go:64] FLAG: --enable-server="true"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234554    2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234559    2550 flags.go:64] FLAG: --event-burst="100"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234562    2550 flags.go:64] FLAG: --event-qps="50"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234566    2550 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234569    2550 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234573    2550 flags.go:64] FLAG: --eviction-hard=""
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234577    2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 20 19:22:28.240608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234580    2550 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234584    2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234587    2550 flags.go:64] FLAG: --eviction-soft=""
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234590    2550 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234593    2550 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234596    2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234599    2550 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234602    2550 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234605    2550 flags.go:64] FLAG: --fail-swap-on="true"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234608    2550 flags.go:64] FLAG: --feature-gates=""
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234612    2550 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234615    2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234618    2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234622    2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234625    2550 flags.go:64] FLAG: --healthz-port="10248"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234630    2550 flags.go:64] FLAG: --help="false"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234633    2550 flags.go:64] FLAG: --hostname-override="ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234636    2550 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234641    2550 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234644    2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234647    2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234651    2550 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234654    2550 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 20 19:22:28.241214 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234657    2550 flags.go:64] FLAG: --image-service-endpoint=""
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234660    2550 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234664    2550 flags.go:64] FLAG: --kube-api-burst="100"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234667    2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234670    2550 flags.go:64] FLAG: --kube-api-qps="50"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234673    2550 flags.go:64] FLAG: --kube-reserved=""
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234676    2550 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234679    2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234683    2550 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234686    2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234689    2550 flags.go:64] FLAG: --lock-file=""
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234692    2550 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234695    2550 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234699    2550 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234705    2550 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234708    2550 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234711    2550 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234714    2550 flags.go:64] FLAG: --logging-format="text"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234717    2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234721    2550 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234724    2550 flags.go:64] FLAG: --manifest-url=""
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234727    2550 flags.go:64] FLAG: --manifest-url-header=""
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234731    2550 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234735    2550 flags.go:64] FLAG: --max-open-files="1000000"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234741    2550 flags.go:64] FLAG: --max-pods="110"
Jul 20 19:22:28.242268 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234744    2550 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234748    2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234752    2550 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234755    2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234758    2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234761    2550 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234764    2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234791    2550 flags.go:64] FLAG: --node-status-max-images="50"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234795    2550 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234798    2550 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234803    2550 flags.go:64] FLAG: --pod-cidr=""
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234806    2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234812    2550 flags.go:64] FLAG: --pod-manifest-path=""
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234815    2550 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234818    2550 flags.go:64] FLAG: --pods-per-core="0"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234822    2550 flags.go:64] FLAG: --port="10250"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234826    2550 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234829    2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07f5864889f4a27a5"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234833    2550 flags.go:64] FLAG: --qos-reserved=""
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234836    2550 flags.go:64] FLAG: --read-only-port="10255"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234839    2550 flags.go:64] FLAG: --register-node="true"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234843    2550 flags.go:64] FLAG: --register-with-taints=""
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234846    2550 flags.go:64] FLAG: --registry-burst="10"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234849    2550 flags.go:64] FLAG: --registry-qps="5"
Jul 20 19:22:28.243421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234852    2550 flags.go:64] FLAG: --reserved-cpus=""
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234855    2550 flags.go:64] FLAG: --reserved-memory=""
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234859    2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234863    2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234866    2550 flags.go:64] FLAG: --rotate-certificates="false"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234869    2550 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234872    2550 flags.go:64] FLAG: --runonce="false"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234876    2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234880    2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234883    2550 flags.go:64] FLAG: --seccomp-default="false"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234886    2550 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234890    2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234893    2550 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234897    2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234900    2550 flags.go:64] FLAG: --storage-driver-password="root"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234903    2550 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234906    2550 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234909    2550 flags.go:64] FLAG: --storage-driver-user="root"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234912    2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234915    2550 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234918    2550 flags.go:64] FLAG: --system-cgroups=""
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234922    2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234927    2550 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234930    2550 flags.go:64] FLAG: --tls-cert-file=""
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234934    2550 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 20 19:22:28.244186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234938    2550 flags.go:64] FLAG: --tls-min-version=""
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234941    2550 flags.go:64] FLAG: --tls-private-key-file=""
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234945    2550 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234948    2550 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234950    2550 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234954    2550 flags.go:64] FLAG: --v="2"
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234957    2550 flags.go:64] FLAG: --version="false"
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234962    2550 flags.go:64] FLAG: --vmodule=""
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234966    2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.234969    2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235092    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235097    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235100    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235103    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235109    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235112    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235115    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235117    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235121    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235125    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235129    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 19:22:28.245198 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235132    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235135    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235138    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235142    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235144    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235147    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235150    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235153    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235155    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235158    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235161    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235164    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235167    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235170    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235173    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235176    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235179    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235181    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235184    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235187    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 19:22:28.246162 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235190    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235192    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235195    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235198    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235202    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235206    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235209    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235212    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235215    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235218    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235221    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235223    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235227    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235230    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235233    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235236    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235239    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235241    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235244    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 19:22:28.246785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235247    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235250    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235253    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235255    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235258    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235261    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235263    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235267    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235269    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235272    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235275    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235279    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235283    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235286    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235289    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235291    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235294    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235297    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235301    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235304    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 19:22:28.247618 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235307    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235310    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235313    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235316    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235318    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235321    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235324    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235327    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235347    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235350    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235353    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235356    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235360    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235364    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235367    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235371    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235374    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235377    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235380    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235382    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 19:22:28.248456 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235386    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235389    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235391    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235394    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235397    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235400    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235402    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235405    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.235408    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.235418    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.243711    2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.243733    2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.244545    2550 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.244565    2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 20 19:22:28.249115 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244661    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244672    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244680    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244686    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244692    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244697    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244702    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244707    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244712    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244717    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244722    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244727    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244732    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244737    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244742    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244747    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244752    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244756    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244761    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 19:22:28.249514 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244768    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244774    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244779    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244785    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244790    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244795    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244801    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244806    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244813    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244820    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244827    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244832    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244838    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244843    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244848    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244853    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244858    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244864    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244869    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 19:22:28.250045 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244873    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244878    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244883    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244888    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244892    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244897    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244901    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244906    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244911    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244916    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244920    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244926    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244930    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244935    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244940    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244944    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244949    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244953    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244959    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244964    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 19:22:28.250554 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244968    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244973    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244977    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244982    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244988    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244993    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.244998    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245002    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245007    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245011    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245016    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245020    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245025    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245029    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245034    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245039    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245043    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245048    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245052    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245057    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 19:22:28.251070 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245061    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245066    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245071    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245076    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245080    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245085    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245089    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245094    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245098    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245103    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245108    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245114    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245118    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245123    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245127    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245132    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245137    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245143    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245147    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245152    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 19:22:28.251666 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245156    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.245165    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245380    2550 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245390    2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245395    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245400    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245406    2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245411    2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245415    2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245420    2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245424    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245429    2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245434    2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245441    2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 20 19:22:28.252194 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245449    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245454    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245459    2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245464    2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245468    2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245473    2550 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245477    2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245481    2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245486    2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245491    2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245496    2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245500    2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245505    2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245510    2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245514    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245518    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245524    2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245528    2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245533    2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245537    2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 20 19:22:28.252580 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245542    2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245547    2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245551    2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245556    2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245561    2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245565    2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245569    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245574    2550 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245579    2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245583    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245588    2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245593    2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245597    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245602    2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245606    2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245610    2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245615    2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245619    2550 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245624    2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 20 19:22:28.253144 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245629    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245633    2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245638    2550 feature_gate.go:328] unrecognized feature gate: Example
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245642    2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245647    2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245651    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245655    2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245662    2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245668    2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245674    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245680    2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245685    2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245690    2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245694    2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245699    2550 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245704    2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245708    2550 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245713    2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245717    2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245722    2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 20 19:22:28.253686 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245726    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245731    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245735    2550 feature_gate.go:328] unrecognized feature gate: Example2
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245740    2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245745    2550 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245750    2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245755    2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245760    2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245764    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245771    2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245777    2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245782    2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245787    2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245793    2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245798    2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245802    2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245807    2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245812    2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245817    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245822    2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 20 19:22:28.254200 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245826    2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245831    2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245836    2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245841    2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245846    2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245850    2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245855    2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:28.245859    2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.245868    2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.246030    2550 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.248891    2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.251200    2550 server.go:1019] "Starting client certificate rotation"
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.251367    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.251409    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 20 19:22:28.255091 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.254908    2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 19:22:28.258645 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.256959    2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 20 19:22:28.268567 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.268552    2550 log.go:25] "Validated CRI v1 runtime API"
Jul 20 19:22:28.270735 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.270719    2550 log.go:25] "Validated CRI v1 image API"
Jul 20 19:22:28.272361 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.272342    2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 20 19:22:28.275674 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.275654    2550 fs.go:135] Filesystem UUIDs: map[51e09b98-6ecf-491b-9326-d76b4a1d8a2b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9db79fe6-accc-41ed-a9c4-4d9f61ad9e07:/dev/nvme0n1p3]
Jul 20 19:22:28.275736 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.275674    2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 20 19:22:28.281078 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.280966    2550 manager.go:216] Machine: {Timestamp:2026-07-20 19:22:28.279656463 +0000 UTC m=+0.365607455 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102835 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec250eb0d09575709d54ae4b0c0d5970 SystemUUID:ec250eb0-d095-7570-9d54-ae4b0c0d5970 BootID:1775f989-5b80-4573-aae0-924fb09d5f7c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:80:e3:c7:d2:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:80:e3:c7:d2:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:f8:91:88:17:a7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 20 19:22:28.281078 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.281076    2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 20 19:22:28.281190 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.281157    2550 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 20 19:22:28.283256 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.283234    2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 20 19:22:28.283411 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.283259    2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-227.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 20 19:22:28.284658 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.284647    2550 topology_manager.go:138] "Creating topology manager with none policy"
Jul 20 19:22:28.284696 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.284663    2550 container_manager_linux.go:309] "Creating device plugin manager"
Jul 20 19:22:28.284696 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.284670    2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 19:22:28.285434 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.285423    2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 20 19:22:28.287115 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.287095    2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 20 19:22:28.290353 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.290324    2550 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 19:22:28.290475 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.290464    2550 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 20 19:22:28.291127 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.291116    2550 kubelet.go:486] "Attempting to sync node with API server"
Jul 20 19:22:28.291167 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.291134    2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 20 19:22:28.291167 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.291150    2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 20 19:22:28.291167 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.291160    2550 kubelet.go:398] "Adding apiserver pod source"
Jul 20 19:22:28.291292 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.291170    2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 20 19:22:28.291561 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.291548    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 19:22:28.291607 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.291570    2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 20 19:22:28.293666 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.293651    2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 20 19:22:28.296084 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.296067    2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 20 19:22:28.296137 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.296110    2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 20 19:22:28.297471 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297447    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 20 19:22:28.297471 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297470    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 20 19:22:28.297594 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297498    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 20 19:22:28.297594 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297508    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 20 19:22:28.297594 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297521    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 20 19:22:28.297594 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297530    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 20 19:22:28.297594 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297543    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 20 19:22:28.297594 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297552    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 20 19:22:28.297594 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297561    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 20 19:22:28.297594 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297570    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 20 19:22:28.297594 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297592    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 20 19:22:28.297897 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.297601    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 20 19:22:28.298725 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.298700    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 20 19:22:28.298725 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.298719    2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 20 19:22:28.300901 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.300879    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 19:22:28.302103 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.302085    2550 server.go:1268] "Started kubelet"
Jul 20 19:22:28.302265 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.302221    2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 20 19:22:28.302368 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.302306    2550 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 20 19:22:28.303325 ip-10-0-138-227 systemd[1]: Started Kubernetes Kubelet.
Jul 20 19:22:28.303815 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.303568    2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 20 19:22:28.306447 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.306406    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 20 19:22:28.306447 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.306413    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 20 19:22:28.306601 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.306521    2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 20 19:22:28.306717 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.306700    2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-227.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 20 19:22:28.308255 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.308233    2550 server.go:310] "Adding debug handlers to kubelet server"
Jul 20 19:22:28.309644 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.309624    2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 20 19:22:28.313541 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.313527    2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 20 19:22:28.313610 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.313543    2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 20 19:22:28.313680 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.313614    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-227.ec2.internal\" not found"
Jul 20 19:22:28.314228 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314207    2550 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 20 19:22:28.314317 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314231    2550 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 20 19:22:28.314317 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314204    2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 20 19:22:28.314444 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314354    2550 reconstruct.go:97] "Volume reconstruction finished"
Jul 20 19:22:28.314444 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314367    2550 reconciler.go:29] "Reconciler: start to sync state"
Jul 20 19:22:28.314543 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314477    2550 factory.go:55] Registering systemd factory
Jul 20 19:22:28.314543 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314517    2550 factory.go:223] Registration of the systemd container factory successfully
Jul 20 19:22:28.314766 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314751    2550 factory.go:153] Registering CRI-O factory
Jul 20 19:22:28.314847 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314770    2550 factory.go:223] Registration of the crio container factory successfully
Jul 20 19:22:28.314902 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314847    2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 20 19:22:28.314902 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314873    2550 factory.go:103] Registering Raw factory
Jul 20 19:22:28.314902 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.314890    2550 manager.go:1196] Started watching for new ooms in manager
Jul 20 19:22:28.315381 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.315364    2550 manager.go:349] Starting recovery of all containers
Jul 20 19:22:28.315886 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.314001    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-227.ec2.internal.18c415d2fe733bd0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-227.ec2.internal,UID:ip-10-0-138-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-227.ec2.internal,},FirstTimestamp:2026-07-20 19:22:28.301839312 +0000 UTC m=+0.387790303,LastTimestamp:2026-07-20 19:22:28.301839312 +0000 UTC m=+0.387790303,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-227.ec2.internal,}"
Jul 20 19:22:28.318396 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.318368    2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 20 19:22:28.318503 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.318459    2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 20 19:22:28.326178 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.326161    2550 manager.go:354] Recovery completed
Jul 20 19:22:28.327647 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.327630    2550 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 20 19:22:28.331104 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.331091    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 19:22:28.332226 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.332207    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5sjlx"
Jul 20 19:22:28.333360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.333343    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 19:22:28.333459 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.333374    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 19:22:28.333459 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.333384    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientPID"
Jul 20 19:22:28.333937 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.333921    2550 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 20 19:22:28.333937 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.333935    2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 20 19:22:28.334075 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.333926    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 19:22:28.334075 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.333953    2550 state_mem.go:36] "Initialized new in-memory state store"
Jul 20 19:22:28.334820 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.334800    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 19:22:28.334914 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.334830    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 19:22:28.334914 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.334842    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientPID"
Jul 20 19:22:28.335280 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.335212    2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-227.ec2.internal.18c415d300542f98  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-227.ec2.internal,UID:ip-10-0-138-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-227.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-227.ec2.internal,},FirstTimestamp:2026-07-20 19:22:28.333359 +0000 UTC m=+0.419309991,LastTimestamp:2026-07-20 19:22:28.333359 +0000 UTC m=+0.419309991,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-227.ec2.internal,}"
Jul 20 19:22:28.335384 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.335367    2550 policy_none.go:49] "None policy: Start"
Jul 20 19:22:28.335493 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.335395    2550 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 20 19:22:28.335493 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.335406    2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 20 19:22:28.336772 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.336759    2550 policy_none.go:47] "Start"
Jul 20 19:22:28.340870 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.340849    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5sjlx"
Jul 20 19:22:28.374916 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.374898    2550 manager.go:337] "Starting Device Plugin manager"
Jul 20 19:22:28.375048 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.374935    2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 20 19:22:28.375048 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.374945    2550 server.go:87] "Starting device plugin registration server"
Jul 20 19:22:28.375244 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.375232    2550 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 20 19:22:28.375297 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.375247    2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 20 19:22:28.375379 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.375354    2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 20 19:22:28.375476 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.375462    2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 20 19:22:28.375531 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.375477    2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 20 19:22:28.376044 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.376019    2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 20 19:22:28.376143 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.376084    2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-227.ec2.internal\" not found"
Jul 20 19:22:28.442753 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.442719    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 20 19:22:28.444114 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.444023    2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 20 19:22:28.444114 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.444062    2550 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 20 19:22:28.444114 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.444084    2550 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 20 19:22:28.444285 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.444122    2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 20 19:22:28.447187 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.447165    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 19:22:28.475572 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.475521    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 19:22:28.476442 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.476417    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 19:22:28.476534 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.476455    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 19:22:28.476534 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.476471    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientPID"
Jul 20 19:22:28.476534 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.476498    2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.483220 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.483201    2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.483301 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.483225    2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-227.ec2.internal\": node \"ip-10-0-138-227.ec2.internal\" not found"
Jul 20 19:22:28.496317 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.496294    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-227.ec2.internal\" not found"
Jul 20 19:22:28.545063 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.545015    2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-227.ec2.internal"]
Jul 20 19:22:28.545212 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.545104    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 19:22:28.546005 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.545991    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 19:22:28.546070 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.546021    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 19:22:28.546070 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.546034    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientPID"
Jul 20 19:22:28.547476 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.547464    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 19:22:28.547638 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.547624    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.547678 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.547655    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 19:22:28.548178 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.548160    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 19:22:28.548250 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.548165    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 19:22:28.548250 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.548193    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 19:22:28.548250 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.548209    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientPID"
Jul 20 19:22:28.548250 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.548210    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 19:22:28.548250 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.548224    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientPID"
Jul 20 19:22:28.549577 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.549560    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.549674 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.549584    2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 20 19:22:28.550218 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.550204    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientMemory"
Jul 20 19:22:28.550332 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.550227    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasNoDiskPressure"
Jul 20 19:22:28.550332 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.550238    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeHasSufficientPID"
Jul 20 19:22:28.580132 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.580112    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-227.ec2.internal\" not found" node="ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.584994 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.584978    2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-227.ec2.internal\" not found" node="ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.596524 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.596504    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-227.ec2.internal\" not found"
Jul 20 19:22:28.697132 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.697105    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-227.ec2.internal\" not found"
Jul 20 19:22:28.716071 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.716046    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c41f37e7a40aac129466553a91e6d57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal\" (UID: \"8c41f37e7a40aac129466553a91e6d57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.716178 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.716080    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c41f37e7a40aac129466553a91e6d57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal\" (UID: \"8c41f37e7a40aac129466553a91e6d57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.716178 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.716106    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2e50227d65fda4050a8a8dd57637707-config\") pod \"kube-apiserver-proxy-ip-10-0-138-227.ec2.internal\" (UID: \"e2e50227d65fda4050a8a8dd57637707\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.797654 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.797587    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-227.ec2.internal\" not found"
Jul 20 19:22:28.816924 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.816897    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c41f37e7a40aac129466553a91e6d57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal\" (UID: \"8c41f37e7a40aac129466553a91e6d57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.816924 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.816843    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c41f37e7a40aac129466553a91e6d57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal\" (UID: \"8c41f37e7a40aac129466553a91e6d57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.817014 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.816963    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c41f37e7a40aac129466553a91e6d57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal\" (UID: \"8c41f37e7a40aac129466553a91e6d57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.817050 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.817014    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c41f37e7a40aac129466553a91e6d57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal\" (UID: \"8c41f37e7a40aac129466553a91e6d57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.817050 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.817032    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2e50227d65fda4050a8a8dd57637707-config\") pod \"kube-apiserver-proxy-ip-10-0-138-227.ec2.internal\" (UID: \"e2e50227d65fda4050a8a8dd57637707\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.817116 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.817061    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2e50227d65fda4050a8a8dd57637707-config\") pod \"kube-apiserver-proxy-ip-10-0-138-227.ec2.internal\" (UID: \"e2e50227d65fda4050a8a8dd57637707\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.887396 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.887359    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.888482 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:28.888466    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:28.898465 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:28.898436    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-227.ec2.internal\" not found"
Jul 20 19:22:29.003531 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.003491    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-227.ec2.internal\" not found"
Jul 20 19:22:29.103921 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.103836    2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-227.ec2.internal\" not found"
Jul 20 19:22:29.162907 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.162882    2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 19:22:29.213883 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.213851    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:29.225882 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.225856    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 19:22:29.227741 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.227723    2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-227.ec2.internal"
Jul 20 19:22:29.235015 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.234990    2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 20 19:22:29.249973 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.249951    2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 20 19:22:29.250092 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.250072    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 19:22:29.250139 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.250120    2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 20 19:22:29.291843 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.291818    2550 apiserver.go:52] "Watching apiserver"
Jul 20 19:22:29.307717 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.307696    2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 20 19:22:29.308154 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.308126    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-6bdk5","openshift-network-diagnostics/network-check-target-qcz7g","openshift-network-operator/iptables-alerter-c4v55","kube-system/konnectivity-agent-mb4tg","kube-system/kube-apiserver-proxy-ip-10-0-138-227.ec2.internal","openshift-dns/node-resolver-2kzc8","openshift-image-registry/node-ca-54b4h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal","openshift-multus/multus-additional-cni-plugins-8m7fr","openshift-ovn-kubernetes/ovnkube-node-42skd","kube-system/global-pull-secret-syncer-ww9zr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz","openshift-cluster-node-tuning-operator/tuned-7xbj9","openshift-multus/multus-55bk8"]
Jul 20 19:22:29.309945 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.309927    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-54b4h"
Jul 20 19:22:29.312152 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.312122    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 20 19:22:29.312280 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.312257    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 20 19:22:29.312367 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.312349    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wvtpr\""
Jul 20 19:22:29.312476 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.312457    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 20 19:22:29.313751 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.313731    2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 20 19:22:29.314830 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.314806    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:29.314921 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.314879    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:29.316571 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.316554    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4v55"
Jul 20 19:22:29.320495 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.318494    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9nbz6\""
Jul 20 19:22:29.320495 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.318583    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 20 19:22:29.320495 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.318817    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 20 19:22:29.320495 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.319416    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 20 19:22:29.320796 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.320740    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:29.321206 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.321181    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8b8de86-0507-4fb5-a218-ef90a0a8100c-host\") pod \"node-ca-54b4h\" (UID: \"d8b8de86-0507-4fb5-a218-ef90a0a8100c\") " pod="openshift-image-registry/node-ca-54b4h"
Jul 20 19:22:29.321313 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.321235    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8b8de86-0507-4fb5-a218-ef90a0a8100c-serviceca\") pod \"node-ca-54b4h\" (UID: \"d8b8de86-0507-4fb5-a218-ef90a0a8100c\") " pod="openshift-image-registry/node-ca-54b4h"
Jul 20 19:22:29.321392 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.321303    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5jhv\" (UniqueName: \"kubernetes.io/projected/d8b8de86-0507-4fb5-a218-ef90a0a8100c-kube-api-access-l5jhv\") pod \"node-ca-54b4h\" (UID: \"d8b8de86-0507-4fb5-a218-ef90a0a8100c\") " pod="openshift-image-registry/node-ca-54b4h"
Jul 20 19:22:29.321392 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.321337    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/57db0719-7a2e-49bf-9356-8b1ea7b94069-kubelet-config\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:29.321392 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.321356    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/57db0719-7a2e-49bf-9356-8b1ea7b94069-dbus\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:29.321392 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.321378    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:29.322688 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.322669    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.323172 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.323153    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 20 19:22:29.324203 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.324157    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2kzc8"
Jul 20 19:22:29.324323 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.324252    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 20 19:22:29.324323 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.324302    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z9htf\""
Jul 20 19:22:29.325940 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.325922    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.327095 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.327074    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 20 19:22:29.327196 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.327112    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 20 19:22:29.327820 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.327476    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pnfwh\""
Jul 20 19:22:29.327820 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.327489    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.327820 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.327492    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 20 19:22:29.327820 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.327696    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 20 19:22:29.327820 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.327697    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 20 19:22:29.328132 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.327871    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 20 19:22:29.328132 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.327986    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 20 19:22:29.328132 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.328053    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-csth2\""
Jul 20 19:22:29.328132 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.328087    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 20 19:22:29.328772 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.328212    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m9td5\""
Jul 20 19:22:29.329347 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.329315    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:29.329444 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.329375    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:29.329444 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.329411    2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 20 19:22:29.330222 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.330201    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 20 19:22:29.330291 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.330229    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wfwhl\""
Jul 20 19:22:29.330291 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.330262    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 20 19:22:29.330910 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.330895    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.332884 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.332869    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:29.332980 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.332924    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:29.333091 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.333078    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 20 19:22:29.333133 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.333099    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 20 19:22:29.333221 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.333204    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 20 19:22:29.333529 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.333513    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 20 19:22:29.333575 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.333548    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 20 19:22:29.333617 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.333585    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cn7w2\""
Jul 20 19:22:29.333617 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.333548    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 20 19:22:29.334966 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.334951    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.338287 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.338253    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wg8m4\""
Jul 20 19:22:29.338287 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.338284    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 20 19:22:29.338456 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.338348    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 20 19:22:29.338771 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.338757    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 20 19:22:29.344897 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.344796    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-19 19:17:28 +0000 UTC" deadline="2028-01-07 23:58:54.799067862 +0000 UTC"
Jul 20 19:22:29.344897 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.344844    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12868h36m25.454228157s"
Jul 20 19:22:29.353391 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.353367    2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9s4dk"
Jul 20 19:22:29.363048 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.363024    2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9s4dk"
Jul 20 19:22:29.415174 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.415141    2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 20 19:22:29.421751 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.421728    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-run-systemd\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.421864 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.421760    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/57db0719-7a2e-49bf-9356-8b1ea7b94069-kubelet-config\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:29.421864 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.421786    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-multus-conf-dir\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.421864 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.421810    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxn6r\" (UniqueName: \"kubernetes.io/projected/ed7bad58-74fb-44fc-9d4d-f2f04d4a1602-kube-api-access-kxn6r\") pod \"node-resolver-2kzc8\" (UID: \"ed7bad58-74fb-44fc-9d4d-f2f04d4a1602\") " pod="openshift-dns/node-resolver-2kzc8"
Jul 20 19:22:29.421864 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.421837    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:29.422089 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.421900    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/57db0719-7a2e-49bf-9356-8b1ea7b94069-kubelet-config\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:29.422089 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.421955    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-run-netns\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.422089 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.421984    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-modprobe-d\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.422089 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422024    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-sys\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.422089 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.422039    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:29.422089 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422050    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-etc-openvswitch\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.422089 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422078    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-run-ovn\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422104    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8b8de86-0507-4fb5-a218-ef90a0a8100c-host\") pod \"node-ca-54b4h\" (UID: \"d8b8de86-0507-4fb5-a218-ef90a0a8100c\") " pod="openshift-image-registry/node-ca-54b4h"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422127    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed7bad58-74fb-44fc-9d4d-f2f04d4a1602-hosts-file\") pod \"node-resolver-2kzc8\" (UID: \"ed7bad58-74fb-44fc-9d4d-f2f04d4a1602\") " pod="openshift-dns/node-resolver-2kzc8"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422150    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kwpx\" (UniqueName: \"kubernetes.io/projected/cd899fed-afcf-484c-89bf-28566c07b2d2-kube-api-access-2kwpx\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422178    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8b8de86-0507-4fb5-a218-ef90a0a8100c-host\") pod \"node-ca-54b4h\" (UID: \"d8b8de86-0507-4fb5-a218-ef90a0a8100c\") " pod="openshift-image-registry/node-ca-54b4h"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.422188    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret podName:57db0719-7a2e-49bf-9356-8b1ea7b94069 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:29.922159671 +0000 UTC m=+2.008110653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret") pod "global-pull-secret-syncer-ww9zr" (UID: "57db0719-7a2e-49bf-9356-8b1ea7b94069") : object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422241    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb7692a4-c508-46e6-90c6-a21fb9450d3a-host-slash\") pod \"iptables-alerter-c4v55\" (UID: \"fb7692a4-c508-46e6-90c6-a21fb9450d3a\") " pod="openshift-network-operator/iptables-alerter-c4v55"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422334    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-systemd\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422355    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m6z6\" (UniqueName: \"kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6\") pod \"network-check-target-qcz7g\" (UID: \"6bcb360a-f63e-47c3-ae1c-eb263e42f691\") " pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422376    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-slash\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422400    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-var-lib-openvswitch\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422423    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-var-lib-kubelet\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422446    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-etc-kubernetes\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.422485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422485    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7rwx\" (UniqueName: \"kubernetes.io/projected/fb7692a4-c508-46e6-90c6-a21fb9450d3a-kube-api-access-h7rwx\") pod \"iptables-alerter-c4v55\" (UID: \"fb7692a4-c508-46e6-90c6-a21fb9450d3a\") " pod="openshift-network-operator/iptables-alerter-c4v55"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422514    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422536    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4531c13-81bf-480d-b18e-0a31aceb78d6-ovn-node-metrics-cert\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422553    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422605    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-sys-fs\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422643    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grxcd\" (UniqueName: \"kubernetes.io/projected/2be025f3-18ef-47df-a00b-d5fcd57bcc72-kube-api-access-grxcd\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422673    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b013675-0f9f-4383-bb33-cf662c795463-cni-binary-copy\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422697    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx86r\" (UniqueName: \"kubernetes.io/projected/0b013675-0f9f-4383-bb33-cf662c795463-kube-api-access-jx86r\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422727    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/394d8ca9-d411-4acf-b1d3-bc7108992e1d-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422746    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-cni-netd\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422762    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmgjs\" (UniqueName: \"kubernetes.io/projected/e4531c13-81bf-480d-b18e-0a31aceb78d6-kube-api-access-kmgjs\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422799    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/57db0719-7a2e-49bf-9356-8b1ea7b94069-dbus\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422829    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4531c13-81bf-480d-b18e-0a31aceb78d6-ovnkube-script-lib\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422846    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-registration-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422908    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8b8de86-0507-4fb5-a218-ef90a0a8100c-serviceca\") pod \"node-ca-54b4h\" (UID: \"d8b8de86-0507-4fb5-a218-ef90a0a8100c\") " pod="openshift-image-registry/node-ca-54b4h"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422952    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-system-cni-dir\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.423041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.422984    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-run-k8s-cni-cncf-io\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423001    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/57db0719-7a2e-49bf-9356-8b1ea7b94069-dbus\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423055    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-os-release\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423077    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlq9j\" (UniqueName: \"kubernetes.io/projected/d51fbd0d-1b64-4a00-9eb3-495652e656ec-kube-api-access-xlq9j\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423101    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-kubelet\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423124    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-multus-cni-dir\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423156    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/394d8ca9-d411-4acf-b1d3-bc7108992e1d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423192    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-sysctl-d\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423224    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-run-openvswitch\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423261    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4531c13-81bf-480d-b18e-0a31aceb78d6-ovnkube-config\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423303    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4531c13-81bf-480d-b18e-0a31aceb78d6-env-overrides\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423336    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b013675-0f9f-4383-bb33-cf662c795463-multus-daemon-config\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423373    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-system-cni-dir\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423397    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8b8de86-0507-4fb5-a218-ef90a0a8100c-serviceca\") pod \"node-ca-54b4h\" (UID: \"d8b8de86-0507-4fb5-a218-ef90a0a8100c\") " pod="openshift-image-registry/node-ca-54b4h"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423412    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d51fbd0d-1b64-4a00-9eb3-495652e656ec-tmp\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423468    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-node-log\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.423812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423494    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423521    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-device-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423549    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-os-release\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423574    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-multus-socket-dir-parent\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423600    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-var-lib-cni-bin\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423623    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-cnibin\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423657    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423689    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-lib-modules\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423735    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-run-netns\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423759    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-log-socket\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423783    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e7a94d4-ebe8-4991-92b5-157d5153b6ef-agent-certs\") pod \"konnectivity-agent-mb4tg\" (UID: \"1e7a94d4-ebe8-4991-92b5-157d5153b6ef\") " pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423810    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e7a94d4-ebe8-4991-92b5-157d5153b6ef-konnectivity-ca\") pod \"konnectivity-agent-mb4tg\" (UID: \"1e7a94d4-ebe8-4991-92b5-157d5153b6ef\") " pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423854    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-host\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423886    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-cni-bin\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423912    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-socket-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423936    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-etc-selinux\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.424552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.423967    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-cnibin\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424012    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/394d8ca9-d411-4acf-b1d3-bc7108992e1d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424041    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bw2\" (UniqueName: \"kubernetes.io/projected/394d8ca9-d411-4acf-b1d3-bc7108992e1d-kube-api-access-67bw2\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424070    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb7692a4-c508-46e6-90c6-a21fb9450d3a-iptables-alerter-script\") pod \"iptables-alerter-c4v55\" (UID: \"fb7692a4-c508-46e6-90c6-a21fb9450d3a\") " pod="openshift-network-operator/iptables-alerter-c4v55"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424091    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-var-lib-kubelet\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424110    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-systemd-units\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424125    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-tuned\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424147    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5jhv\" (UniqueName: \"kubernetes.io/projected/d8b8de86-0507-4fb5-a218-ef90a0a8100c-kube-api-access-l5jhv\") pod \"node-ca-54b4h\" (UID: \"d8b8de86-0507-4fb5-a218-ef90a0a8100c\") " pod="openshift-image-registry/node-ca-54b4h"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424171    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-var-lib-cni-multus\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424200    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-hostroot\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424228    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-run-multus-certs\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424260    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-sysctl-conf\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424309    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-run\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424336    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed7bad58-74fb-44fc-9d4d-f2f04d4a1602-tmp-dir\") pod \"node-resolver-2kzc8\" (UID: \"ed7bad58-74fb-44fc-9d4d-f2f04d4a1602\") " pod="openshift-dns/node-resolver-2kzc8"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424352    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424376    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-sysconfig\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.425147 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.424401    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-kubernetes\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.431986 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.431968    2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 19:22:29.433969 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.433949    2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 20 19:22:29.437575 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.437552    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5jhv\" (UniqueName: \"kubernetes.io/projected/d8b8de86-0507-4fb5-a218-ef90a0a8100c-kube-api-access-l5jhv\") pod \"node-ca-54b4h\" (UID: \"d8b8de86-0507-4fb5-a218-ef90a0a8100c\") " pod="openshift-image-registry/node-ca-54b4h"
Jul 20 19:22:29.444311 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:29.444282    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e50227d65fda4050a8a8dd57637707.slice/crio-e1fa5a0eeb46316f515cb7e59b4104fe793702c69916e3c8aa09af63430605d7 WatchSource:0}: Error finding container e1fa5a0eeb46316f515cb7e59b4104fe793702c69916e3c8aa09af63430605d7: Status 404 returned error can't find the container with id e1fa5a0eeb46316f515cb7e59b4104fe793702c69916e3c8aa09af63430605d7
Jul 20 19:22:29.444531 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:29.444510    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c41f37e7a40aac129466553a91e6d57.slice/crio-8aff6d5def4807b855bc63b19afca18af2ff98642b9bc40a7b94b30299e21b3e WatchSource:0}: Error finding container 8aff6d5def4807b855bc63b19afca18af2ff98642b9bc40a7b94b30299e21b3e: Status 404 returned error can't find the container with id 8aff6d5def4807b855bc63b19afca18af2ff98642b9bc40a7b94b30299e21b3e
Jul 20 19:22:29.451536 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.451517    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 19:22:29.451927 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.451904    2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 19:22:29.524988 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.524958    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-multus-cni-dir\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.525117 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.524992    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/394d8ca9-d411-4acf-b1d3-bc7108992e1d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.525117 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525010    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-sysctl-d\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.525117 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525026    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-run-openvswitch\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.525117 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525042    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4531c13-81bf-480d-b18e-0a31aceb78d6-ovnkube-config\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.525117 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525059    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4531c13-81bf-480d-b18e-0a31aceb78d6-env-overrides\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.525117 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525108    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-run-openvswitch\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525111    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-multus-cni-dir\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525136    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-sysctl-d\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525163    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b013675-0f9f-4383-bb33-cf662c795463-multus-daemon-config\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525190    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-system-cni-dir\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525206    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d51fbd0d-1b64-4a00-9eb3-495652e656ec-tmp\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525221    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-node-log\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525241    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525268    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-device-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525295    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-system-cni-dir\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-os-release\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525330    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-node-log\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525345    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-multus-socket-dir-parent\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525350    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525386    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-var-lib-cni-bin\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525396    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-multus-socket-dir-parent\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525420    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-cnibin\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.525457 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525430    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-device-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525448    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525456    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-os-release\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525495    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-var-lib-cni-bin\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525499    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-cnibin\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525514    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-lib-modules\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525542    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-run-netns\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525568    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-log-socket\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525576    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4531c13-81bf-480d-b18e-0a31aceb78d6-env-overrides\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525595    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e7a94d4-ebe8-4991-92b5-157d5153b6ef-agent-certs\") pod \"konnectivity-agent-mb4tg\" (UID: \"1e7a94d4-ebe8-4991-92b5-157d5153b6ef\") " pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525600    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-lib-modules\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525621    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e7a94d4-ebe8-4991-92b5-157d5153b6ef-konnectivity-ca\") pod \"konnectivity-agent-mb4tg\" (UID: \"1e7a94d4-ebe8-4991-92b5-157d5153b6ef\") " pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525624    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-run-netns\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525646    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-log-socket\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525647    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525647    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-host\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525685    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-cni-bin\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525696    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-host\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.526246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525712    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-socket-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525738    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-etc-selinux\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525746    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-cni-bin\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525745    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/394d8ca9-d411-4acf-b1d3-bc7108992e1d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525767    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-cnibin\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525793    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/394d8ca9-d411-4acf-b1d3-bc7108992e1d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525801    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b013675-0f9f-4383-bb33-cf662c795463-multus-daemon-config\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525818    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bw2\" (UniqueName: \"kubernetes.io/projected/394d8ca9-d411-4acf-b1d3-bc7108992e1d-kube-api-access-67bw2\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525842    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-etc-selinux\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525847    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb7692a4-c508-46e6-90c6-a21fb9450d3a-iptables-alerter-script\") pod \"iptables-alerter-c4v55\" (UID: \"fb7692a4-c508-46e6-90c6-a21fb9450d3a\") " pod="openshift-network-operator/iptables-alerter-c4v55"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525850    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-socket-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525874    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-cnibin\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525879    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-var-lib-kubelet\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525917    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-systemd-units\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525946    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-tuned\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525974    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-var-lib-cni-multus\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526001    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-hostroot\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.527093 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526008    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-systemd-units\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526017    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4531c13-81bf-480d-b18e-0a31aceb78d6-ovnkube-config\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526027    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-run-multus-certs\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.525920    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-var-lib-kubelet\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526054    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-sysctl-conf\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526081    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-run\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526092    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-hostroot\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526106    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed7bad58-74fb-44fc-9d4d-f2f04d4a1602-tmp-dir\") pod \"node-resolver-2kzc8\" (UID: \"ed7bad58-74fb-44fc-9d4d-f2f04d4a1602\") " pod="openshift-dns/node-resolver-2kzc8"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526571    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526632    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-sysconfig\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526667    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-kubernetes\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526702    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-run-systemd\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526744    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-multus-conf-dir\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526773    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxn6r\" (UniqueName: \"kubernetes.io/projected/ed7bad58-74fb-44fc-9d4d-f2f04d4a1602-kube-api-access-kxn6r\") pod \"node-resolver-2kzc8\" (UID: \"ed7bad58-74fb-44fc-9d4d-f2f04d4a1602\") " pod="openshift-dns/node-resolver-2kzc8"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526830    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-run-netns\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526862    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-modprobe-d\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526889    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-sys\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526922    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-etc-openvswitch\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.527934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526956    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-run-ovn\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527001    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed7bad58-74fb-44fc-9d4d-f2f04d4a1602-hosts-file\") pod \"node-resolver-2kzc8\" (UID: \"ed7bad58-74fb-44fc-9d4d-f2f04d4a1602\") " pod="openshift-dns/node-resolver-2kzc8"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527038    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kwpx\" (UniqueName: \"kubernetes.io/projected/cd899fed-afcf-484c-89bf-28566c07b2d2-kube-api-access-2kwpx\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527068    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb7692a4-c508-46e6-90c6-a21fb9450d3a-host-slash\") pod \"iptables-alerter-c4v55\" (UID: \"fb7692a4-c508-46e6-90c6-a21fb9450d3a\") " pod="openshift-network-operator/iptables-alerter-c4v55"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527104    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-systemd\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527140    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6z6\" (UniqueName: \"kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6\") pod \"network-check-target-qcz7g\" (UID: \"6bcb360a-f63e-47c3-ae1c-eb263e42f691\") " pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527174    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-slash\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527217    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-var-lib-openvswitch\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.526721    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e7a94d4-ebe8-4991-92b5-157d5153b6ef-konnectivity-ca\") pod \"konnectivity-agent-mb4tg\" (UID: \"1e7a94d4-ebe8-4991-92b5-157d5153b6ef\") " pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527234    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/394d8ca9-d411-4acf-b1d3-bc7108992e1d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527252    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-var-lib-kubelet\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527277    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-var-lib-cni-multus\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527287    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-etc-kubernetes\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.527663    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fb7692a4-c508-46e6-90c6-a21fb9450d3a-iptables-alerter-script\") pod \"iptables-alerter-c4v55\" (UID: \"fb7692a4-c508-46e6-90c6-a21fb9450d3a\") " pod="openshift-network-operator/iptables-alerter-c4v55"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528268    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7rwx\" (UniqueName: \"kubernetes.io/projected/fb7692a4-c508-46e6-90c6-a21fb9450d3a-kube-api-access-h7rwx\") pod \"iptables-alerter-c4v55\" (UID: \"fb7692a4-c508-46e6-90c6-a21fb9450d3a\") " pod="openshift-network-operator/iptables-alerter-c4v55"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528366    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-etc-kubernetes\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528377    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-run-multus-certs\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.528809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528437    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-multus-conf-dir\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528441    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-run-systemd\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528472    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-slash\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528491    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-var-lib-openvswitch\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528550    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-var-lib-kubelet\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528572    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-systemd\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528589    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d51fbd0d-1b64-4a00-9eb3-495652e656ec-tmp\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528634    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-run-netns\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528693    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e7a94d4-ebe8-4991-92b5-157d5153b6ef-agent-certs\") pod \"konnectivity-agent-mb4tg\" (UID: \"1e7a94d4-ebe8-4991-92b5-157d5153b6ef\") " pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528698    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fb7692a4-c508-46e6-90c6-a21fb9450d3a-host-slash\") pod \"iptables-alerter-c4v55\" (UID: \"fb7692a4-c508-46e6-90c6-a21fb9450d3a\") " pod="openshift-network-operator/iptables-alerter-c4v55"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528705    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-sysctl-conf\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528701    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed7bad58-74fb-44fc-9d4d-f2f04d4a1602-hosts-file\") pod \"node-resolver-2kzc8\" (UID: \"ed7bad58-74fb-44fc-9d4d-f2f04d4a1602\") " pod="openshift-dns/node-resolver-2kzc8"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528578    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-modprobe-d\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528752    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-run-ovn\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528764    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528824    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-etc-openvswitch\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528943    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4531c13-81bf-480d-b18e-0a31aceb78d6-ovn-node-metrics-cert\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.528980    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.529675 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529002    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529019    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-sys-fs\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529083    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-sys-fs\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529064    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grxcd\" (UniqueName: \"kubernetes.io/projected/2be025f3-18ef-47df-a00b-d5fcd57bcc72-kube-api-access-grxcd\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529141    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b013675-0f9f-4383-bb33-cf662c795463-cni-binary-copy\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529153    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529181    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx86r\" (UniqueName: \"kubernetes.io/projected/0b013675-0f9f-4383-bb33-cf662c795463-kube-api-access-jx86r\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529202    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed7bad58-74fb-44fc-9d4d-f2f04d4a1602-tmp-dir\") pod \"node-resolver-2kzc8\" (UID: \"ed7bad58-74fb-44fc-9d4d-f2f04d4a1602\") " pod="openshift-dns/node-resolver-2kzc8"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529221    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-kubernetes\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529220    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/394d8ca9-d411-4acf-b1d3-bc7108992e1d-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529747    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b013675-0f9f-4383-bb33-cf662c795463-cni-binary-copy\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529806    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-cni-netd\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529851    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmgjs\" (UniqueName: \"kubernetes.io/projected/e4531c13-81bf-480d-b18e-0a31aceb78d6-kube-api-access-kmgjs\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529889    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4531c13-81bf-480d-b18e-0a31aceb78d6-ovnkube-script-lib\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.529927    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-registration-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.530004    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-cni-netd\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.530044    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-system-cni-dir\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.530596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.530083    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-run-k8s-cni-cncf-io\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.531437 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.530121    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-os-release\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.531437 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.530156    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlq9j\" (UniqueName: \"kubernetes.io/projected/d51fbd0d-1b64-4a00-9eb3-495652e656ec-kube-api-access-xlq9j\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.531437 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.530189    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-kubelet\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.531437 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.530286    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4531c13-81bf-480d-b18e-0a31aceb78d6-host-kubelet\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.531437 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.530374    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2be025f3-18ef-47df-a00b-d5fcd57bcc72-registration-dir\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.531437 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.530687    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-system-cni-dir\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.531437 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.530755    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b013675-0f9f-4383-bb33-cf662c795463-host-run-k8s-cni-cncf-io\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.531437 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.530832    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/394d8ca9-d411-4acf-b1d3-bc7108992e1d-os-release\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.531437 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.531111    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-sys\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.531437 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.531185    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:29.531785 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.531559    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-tuned\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.531785 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.531595    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs podName:cd899fed-afcf-484c-89bf-28566c07b2d2 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:30.031238909 +0000 UTC m=+2.117189909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs") pod "network-metrics-daemon-6bdk5" (UID: "cd899fed-afcf-484c-89bf-28566c07b2d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:29.531785 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.531657    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-etc-sysconfig\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.531785 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.531727    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d51fbd0d-1b64-4a00-9eb3-495652e656ec-run\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.531785 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.531570    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4531c13-81bf-480d-b18e-0a31aceb78d6-ovn-node-metrics-cert\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.532140 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.532115    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4531c13-81bf-480d-b18e-0a31aceb78d6-ovnkube-script-lib\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.532775 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.532749    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/394d8ca9-d411-4acf-b1d3-bc7108992e1d-cni-binary-copy\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.540317 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.540300    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 19:22:29.540391 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.540320    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 19:22:29.540391 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.540331    2550 projected.go:196] Error preparing data for projected volume kube-api-access-7m6z6 for pod openshift-network-diagnostics/network-check-target-qcz7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:29.540464 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.540395    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6 podName:6bcb360a-f63e-47c3-ae1c-eb263e42f691 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:30.040383129 +0000 UTC m=+2.126334107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7m6z6" (UniqueName: "kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6") pod "network-check-target-qcz7g" (UID: "6bcb360a-f63e-47c3-ae1c-eb263e42f691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:29.543173 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.543138    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx86r\" (UniqueName: \"kubernetes.io/projected/0b013675-0f9f-4383-bb33-cf662c795463-kube-api-access-jx86r\") pod \"multus-55bk8\" (UID: \"0b013675-0f9f-4383-bb33-cf662c795463\") " pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.543345 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.543304    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bw2\" (UniqueName: \"kubernetes.io/projected/394d8ca9-d411-4acf-b1d3-bc7108992e1d-kube-api-access-67bw2\") pod \"multus-additional-cni-plugins-8m7fr\" (UID: \"394d8ca9-d411-4acf-b1d3-bc7108992e1d\") " pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.543503 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.543484    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7rwx\" (UniqueName: \"kubernetes.io/projected/fb7692a4-c508-46e6-90c6-a21fb9450d3a-kube-api-access-h7rwx\") pod \"iptables-alerter-c4v55\" (UID: \"fb7692a4-c508-46e6-90c6-a21fb9450d3a\") " pod="openshift-network-operator/iptables-alerter-c4v55"
Jul 20 19:22:29.543630 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.543605    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxn6r\" (UniqueName: \"kubernetes.io/projected/ed7bad58-74fb-44fc-9d4d-f2f04d4a1602-kube-api-access-kxn6r\") pod \"node-resolver-2kzc8\" (UID: \"ed7bad58-74fb-44fc-9d4d-f2f04d4a1602\") " pod="openshift-dns/node-resolver-2kzc8"
Jul 20 19:22:29.544448 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.544422    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kwpx\" (UniqueName: \"kubernetes.io/projected/cd899fed-afcf-484c-89bf-28566c07b2d2-kube-api-access-2kwpx\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:29.545037 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.545017    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlq9j\" (UniqueName: \"kubernetes.io/projected/d51fbd0d-1b64-4a00-9eb3-495652e656ec-kube-api-access-xlq9j\") pod \"tuned-7xbj9\" (UID: \"d51fbd0d-1b64-4a00-9eb3-495652e656ec\") " pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.545365 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.545323    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grxcd\" (UniqueName: \"kubernetes.io/projected/2be025f3-18ef-47df-a00b-d5fcd57bcc72-kube-api-access-grxcd\") pod \"aws-ebs-csi-driver-node-k4nvz\" (UID: \"2be025f3-18ef-47df-a00b-d5fcd57bcc72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.545414 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.545397    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmgjs\" (UniqueName: \"kubernetes.io/projected/e4531c13-81bf-480d-b18e-0a31aceb78d6-kube-api-access-kmgjs\") pod \"ovnkube-node-42skd\" (UID: \"e4531c13-81bf-480d-b18e-0a31aceb78d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.643800 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.643704    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-54b4h"
Jul 20 19:22:29.653080 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:29.653051    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b8de86_0507_4fb5_a218_ef90a0a8100c.slice/crio-c0845ec20633796107ab4e34590d3e576a7c6bd76839fbc4b05fec52e00735f4 WatchSource:0}: Error finding container c0845ec20633796107ab4e34590d3e576a7c6bd76839fbc4b05fec52e00735f4: Status 404 returned error can't find the container with id c0845ec20633796107ab4e34590d3e576a7c6bd76839fbc4b05fec52e00735f4
Jul 20 19:22:29.660016 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.659994    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4v55"
Jul 20 19:22:29.668382 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:29.668355    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb7692a4_c508_46e6_90c6_a21fb9450d3a.slice/crio-cc388c5726434afe8457ac58a3a799df509bd80b26cd286d2d279c802e8015b1 WatchSource:0}: Error finding container cc388c5726434afe8457ac58a3a799df509bd80b26cd286d2d279c802e8015b1: Status 404 returned error can't find the container with id cc388c5726434afe8457ac58a3a799df509bd80b26cd286d2d279c802e8015b1
Jul 20 19:22:29.674487 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.674469    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:29.677413 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.677390    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-55bk8"
Jul 20 19:22:29.681625 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:29.681598    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7a94d4_ebe8_4991_92b5_157d5153b6ef.slice/crio-ea112c2efb5e4d2bd38e33751f1c46306fb9293d006428fc4dd3dd6365d692c8 WatchSource:0}: Error finding container ea112c2efb5e4d2bd38e33751f1c46306fb9293d006428fc4dd3dd6365d692c8: Status 404 returned error can't find the container with id ea112c2efb5e4d2bd38e33751f1c46306fb9293d006428fc4dd3dd6365d692c8
Jul 20 19:22:29.685302 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:29.685275    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b013675_0f9f_4383_bb33_cf662c795463.slice/crio-bd151ba3e9d9ca923a28cd3efb3cef4d1c7ceb6f55d021600ffa0685708bb3de WatchSource:0}: Error finding container bd151ba3e9d9ca923a28cd3efb3cef4d1c7ceb6f55d021600ffa0685708bb3de: Status 404 returned error can't find the container with id bd151ba3e9d9ca923a28cd3efb3cef4d1c7ceb6f55d021600ffa0685708bb3de
Jul 20 19:22:29.694373 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.694355    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2kzc8"
Jul 20 19:22:29.701911 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.701891    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8m7fr"
Jul 20 19:22:29.702195 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:29.702175    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7bad58_74fb_44fc_9d4d_f2f04d4a1602.slice/crio-f114eb8875eac9f0d3cb836bf88126801cee35273f83fcce1e722652c6ab03fd WatchSource:0}: Error finding container f114eb8875eac9f0d3cb836bf88126801cee35273f83fcce1e722652c6ab03fd: Status 404 returned error can't find the container with id f114eb8875eac9f0d3cb836bf88126801cee35273f83fcce1e722652c6ab03fd
Jul 20 19:22:29.710038 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:29.710015    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394d8ca9_d411_4acf_b1d3_bc7108992e1d.slice/crio-f22c407a5c0a49d49d3090359b6541a717aa80d71e65d9e465a04c1181959caa WatchSource:0}: Error finding container f22c407a5c0a49d49d3090359b6541a717aa80d71e65d9e465a04c1181959caa: Status 404 returned error can't find the container with id f22c407a5c0a49d49d3090359b6541a717aa80d71e65d9e465a04c1181959caa
Jul 20 19:22:29.721654 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.721635    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7xbj9"
Jul 20 19:22:29.728740 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:29.728719    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51fbd0d_1b64_4a00_9eb3_495652e656ec.slice/crio-23b5df63aa6f811785f5c7185dcef00d1268f16f605fc66b07b43a90fb875210 WatchSource:0}: Error finding container 23b5df63aa6f811785f5c7185dcef00d1268f16f605fc66b07b43a90fb875210: Status 404 returned error can't find the container with id 23b5df63aa6f811785f5c7185dcef00d1268f16f605fc66b07b43a90fb875210
Jul 20 19:22:29.741384 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.741364    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:29.746462 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.746443    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz"
Jul 20 19:22:29.749063 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:29.749045    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4531c13_81bf_480d_b18e_0a31aceb78d6.slice/crio-2e908d4e1a466aeb44c93a32e7c16500a75cabed7843842a03f516c2f7d5def3 WatchSource:0}: Error finding container 2e908d4e1a466aeb44c93a32e7c16500a75cabed7843842a03f516c2f7d5def3: Status 404 returned error can't find the container with id 2e908d4e1a466aeb44c93a32e7c16500a75cabed7843842a03f516c2f7d5def3
Jul 20 19:22:29.753974 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:22:29.753954    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be025f3_18ef_47df_a00b_d5fcd57bcc72.slice/crio-42085bf2e493365572ebeb2728ad29ce6ddec54e55c925bba769fb2b0ed8ea59 WatchSource:0}: Error finding container 42085bf2e493365572ebeb2728ad29ce6ddec54e55c925bba769fb2b0ed8ea59: Status 404 returned error can't find the container with id 42085bf2e493365572ebeb2728ad29ce6ddec54e55c925bba769fb2b0ed8ea59
Jul 20 19:22:29.933706 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:29.933619    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:29.933868 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.933799    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:29.933921 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:29.933880    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret podName:57db0719-7a2e-49bf-9356-8b1ea7b94069 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:30.933857861 +0000 UTC m=+3.019808854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret") pod "global-pull-secret-syncer-ww9zr" (UID: "57db0719-7a2e-49bf-9356-8b1ea7b94069") : object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:30.034672 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.034623    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:30.034873 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:30.034811    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:30.034935 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:30.034889    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs podName:cd899fed-afcf-484c-89bf-28566c07b2d2 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:31.034867321 +0000 UTC m=+3.120818298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs") pod "network-metrics-daemon-6bdk5" (UID: "cd899fed-afcf-484c-89bf-28566c07b2d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:30.135656 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.135573    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6z6\" (UniqueName: \"kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6\") pod \"network-check-target-qcz7g\" (UID: \"6bcb360a-f63e-47c3-ae1c-eb263e42f691\") " pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:30.135854 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:30.135781    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 19:22:30.135854 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:30.135802    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 19:22:30.135854 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:30.135815    2550 projected.go:196] Error preparing data for projected volume kube-api-access-7m6z6 for pod openshift-network-diagnostics/network-check-target-qcz7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:30.136026 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:30.135906    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6 podName:6bcb360a-f63e-47c3-ae1c-eb263e42f691 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:31.135857971 +0000 UTC m=+3.221808950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m6z6" (UniqueName: "kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6") pod "network-check-target-qcz7g" (UID: "6bcb360a-f63e-47c3-ae1c-eb263e42f691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:30.365577 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.365462    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 19:17:29 +0000 UTC" deadline="2028-03-26 03:49:35.146808921 +0000 UTC"
Jul 20 19:22:30.365577 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.365531    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14744h27m4.781281707s"
Jul 20 19:22:30.447747 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.445363    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:30.447747 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:30.445511    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:30.483043 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.482911    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2kzc8" event={"ID":"ed7bad58-74fb-44fc-9d4d-f2f04d4a1602","Type":"ContainerStarted","Data":"f114eb8875eac9f0d3cb836bf88126801cee35273f83fcce1e722652c6ab03fd"}
Jul 20 19:22:30.493686 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.493638    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-55bk8" event={"ID":"0b013675-0f9f-4383-bb33-cf662c795463","Type":"ContainerStarted","Data":"bd151ba3e9d9ca923a28cd3efb3cef4d1c7ceb6f55d021600ffa0685708bb3de"}
Jul 20 19:22:30.506568 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.506534    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mb4tg" event={"ID":"1e7a94d4-ebe8-4991-92b5-157d5153b6ef","Type":"ContainerStarted","Data":"ea112c2efb5e4d2bd38e33751f1c46306fb9293d006428fc4dd3dd6365d692c8"}
Jul 20 19:22:30.523668 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.523642    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4v55" event={"ID":"fb7692a4-c508-46e6-90c6-a21fb9450d3a","Type":"ContainerStarted","Data":"cc388c5726434afe8457ac58a3a799df509bd80b26cd286d2d279c802e8015b1"}
Jul 20 19:22:30.532638 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.532609    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal" event={"ID":"8c41f37e7a40aac129466553a91e6d57","Type":"ContainerStarted","Data":"8aff6d5def4807b855bc63b19afca18af2ff98642b9bc40a7b94b30299e21b3e"}
Jul 20 19:22:30.539385 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.539358    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz" event={"ID":"2be025f3-18ef-47df-a00b-d5fcd57bcc72","Type":"ContainerStarted","Data":"42085bf2e493365572ebeb2728ad29ce6ddec54e55c925bba769fb2b0ed8ea59"}
Jul 20 19:22:30.547568 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.547542    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42skd" event={"ID":"e4531c13-81bf-480d-b18e-0a31aceb78d6","Type":"ContainerStarted","Data":"2e908d4e1a466aeb44c93a32e7c16500a75cabed7843842a03f516c2f7d5def3"}
Jul 20 19:22:30.549718 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.549689    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xbj9" event={"ID":"d51fbd0d-1b64-4a00-9eb3-495652e656ec","Type":"ContainerStarted","Data":"23b5df63aa6f811785f5c7185dcef00d1268f16f605fc66b07b43a90fb875210"}
Jul 20 19:22:30.571475 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.571444    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m7fr" event={"ID":"394d8ca9-d411-4acf-b1d3-bc7108992e1d","Type":"ContainerStarted","Data":"f22c407a5c0a49d49d3090359b6541a717aa80d71e65d9e465a04c1181959caa"}
Jul 20 19:22:30.588830 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.588639    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-54b4h" event={"ID":"d8b8de86-0507-4fb5-a218-ef90a0a8100c","Type":"ContainerStarted","Data":"c0845ec20633796107ab4e34590d3e576a7c6bd76839fbc4b05fec52e00735f4"}
Jul 20 19:22:30.606735 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.606701    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-227.ec2.internal" event={"ID":"e2e50227d65fda4050a8a8dd57637707","Type":"ContainerStarted","Data":"e1fa5a0eeb46316f515cb7e59b4104fe793702c69916e3c8aa09af63430605d7"}
Jul 20 19:22:30.736247 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.736156    2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 19:22:30.942099 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:30.942058    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:30.942291 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:30.942238    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:30.942376 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:30.942326    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret podName:57db0719-7a2e-49bf-9356-8b1ea7b94069 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:32.942286912 +0000 UTC m=+5.028237996 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret") pod "global-pull-secret-syncer-ww9zr" (UID: "57db0719-7a2e-49bf-9356-8b1ea7b94069") : object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:31.043521 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:31.043421    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:31.043706 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:31.043573    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:31.043706 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:31.043645    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs podName:cd899fed-afcf-484c-89bf-28566c07b2d2 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:33.043624232 +0000 UTC m=+5.129575209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs") pod "network-metrics-daemon-6bdk5" (UID: "cd899fed-afcf-484c-89bf-28566c07b2d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:31.145539 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:31.145497    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6z6\" (UniqueName: \"kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6\") pod \"network-check-target-qcz7g\" (UID: \"6bcb360a-f63e-47c3-ae1c-eb263e42f691\") " pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:31.145751 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:31.145735    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 19:22:31.145830 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:31.145760    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 19:22:31.145830 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:31.145775    2550 projected.go:196] Error preparing data for projected volume kube-api-access-7m6z6 for pod openshift-network-diagnostics/network-check-target-qcz7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:31.145950 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:31.145832    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6 podName:6bcb360a-f63e-47c3-ae1c-eb263e42f691 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:33.145814482 +0000 UTC m=+5.231765462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m6z6" (UniqueName: "kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6") pod "network-check-target-qcz7g" (UID: "6bcb360a-f63e-47c3-ae1c-eb263e42f691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:31.366823 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:31.366700    2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-19 19:17:29 +0000 UTC" deadline="2028-02-25 16:26:56.428942962 +0000 UTC"
Jul 20 19:22:31.366823 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:31.366742    2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14037h4m25.062205634s"
Jul 20 19:22:31.445836 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:31.444949    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:31.445836 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:31.445114    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:31.445836 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:31.445562    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:31.445836 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:31.445664    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:31.454163 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:31.453953    2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 20 19:22:32.449040 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:32.448461    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:32.449040 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:32.448597    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:32.962674 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:32.962085    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:32.962674 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:32.962240    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:32.962674 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:32.962317    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret podName:57db0719-7a2e-49bf-9356-8b1ea7b94069 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:36.96229885 +0000 UTC m=+9.048249829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret") pod "global-pull-secret-syncer-ww9zr" (UID: "57db0719-7a2e-49bf-9356-8b1ea7b94069") : object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:33.063781 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:33.063309    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:33.063781 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:33.063482    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:33.063781 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:33.063539    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs podName:cd899fed-afcf-484c-89bf-28566c07b2d2 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:37.063521159 +0000 UTC m=+9.149472140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs") pod "network-metrics-daemon-6bdk5" (UID: "cd899fed-afcf-484c-89bf-28566c07b2d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:33.164870 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:33.164264    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6z6\" (UniqueName: \"kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6\") pod \"network-check-target-qcz7g\" (UID: \"6bcb360a-f63e-47c3-ae1c-eb263e42f691\") " pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:33.164870 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:33.164440    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 19:22:33.164870 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:33.164462    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 19:22:33.164870 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:33.164474    2550 projected.go:196] Error preparing data for projected volume kube-api-access-7m6z6 for pod openshift-network-diagnostics/network-check-target-qcz7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:33.164870 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:33.164528    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6 podName:6bcb360a-f63e-47c3-ae1c-eb263e42f691 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:37.164510196 +0000 UTC m=+9.250461176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m6z6" (UniqueName: "kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6") pod "network-check-target-qcz7g" (UID: "6bcb360a-f63e-47c3-ae1c-eb263e42f691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:33.444494 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:33.444414    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:33.444651 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:33.444434    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:33.444651 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:33.444593    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:33.445099 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:33.445044    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:34.445534 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:34.445500    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:34.446006 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:34.445645    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:35.445283 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:35.445194    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:35.445429 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:35.445337    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:35.445429 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:35.445396    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:35.445544 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:35.445526    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:36.444771 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:36.444737    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:36.445223 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:36.444875    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:36.998860 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:36.998823    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:36.999052 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:36.998971    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:36.999052 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:36.999034    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret podName:57db0719-7a2e-49bf-9356-8b1ea7b94069 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:44.999021186 +0000 UTC m=+17.084972168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret") pod "global-pull-secret-syncer-ww9zr" (UID: "57db0719-7a2e-49bf-9356-8b1ea7b94069") : object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:37.099656 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:37.099550    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:37.100434 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:37.099979    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:37.100434 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:37.100060    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs podName:cd899fed-afcf-484c-89bf-28566c07b2d2 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:45.100039269 +0000 UTC m=+17.185990250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs") pod "network-metrics-daemon-6bdk5" (UID: "cd899fed-afcf-484c-89bf-28566c07b2d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:37.200175 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:37.200135    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6z6\" (UniqueName: \"kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6\") pod \"network-check-target-qcz7g\" (UID: \"6bcb360a-f63e-47c3-ae1c-eb263e42f691\") " pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:37.200365 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:37.200327    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 19:22:37.200365 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:37.200345    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 19:22:37.200365 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:37.200354    2550 projected.go:196] Error preparing data for projected volume kube-api-access-7m6z6 for pod openshift-network-diagnostics/network-check-target-qcz7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:37.200526 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:37.200408    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6 podName:6bcb360a-f63e-47c3-ae1c-eb263e42f691 nodeName:}" failed. No retries permitted until 2026-07-20 19:22:45.200390904 +0000 UTC m=+17.286341897 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m6z6" (UniqueName: "kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6") pod "network-check-target-qcz7g" (UID: "6bcb360a-f63e-47c3-ae1c-eb263e42f691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:37.444887 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:37.444711    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:37.444887 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:37.444733    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:37.444887 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:37.444856    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:37.445632 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:37.445286    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:38.445642 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:38.445601    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:38.446097 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:38.445731    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:39.445149 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:39.444512    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:39.445149 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:39.444647    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:39.445149 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:39.445019    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:39.445149 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:39.445144    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:40.444401 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:40.444341    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:40.444842 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:40.444473    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:41.444695 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:41.444657    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:41.444695 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:41.444702    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:41.445093 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:41.444791    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:41.445093 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:41.444901    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:42.447040 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:42.447013    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:42.447496 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:42.447126    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:43.445174 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:43.445115    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:43.445391 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:43.445244    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:43.445391 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:43.445299    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:43.445509 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:43.445425    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:44.444656 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:44.444612    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:44.445033 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:44.444759    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:45.058146 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:45.058106    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:45.058401 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:45.058293    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:45.058401 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:45.058362    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret podName:57db0719-7a2e-49bf-9356-8b1ea7b94069 nodeName:}" failed. No retries permitted until 2026-07-20 19:23:01.058342325 +0000 UTC m=+33.144293301 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret") pod "global-pull-secret-syncer-ww9zr" (UID: "57db0719-7a2e-49bf-9356-8b1ea7b94069") : object "kube-system"/"original-pull-secret" not registered
Jul 20 19:22:45.158879 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:45.158839    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:45.161499 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:45.159377    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:45.161499 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:45.159483    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs podName:cd899fed-afcf-484c-89bf-28566c07b2d2 nodeName:}" failed. No retries permitted until 2026-07-20 19:23:01.159460093 +0000 UTC m=+33.245411073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs") pod "network-metrics-daemon-6bdk5" (UID: "cd899fed-afcf-484c-89bf-28566c07b2d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:22:45.260104 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:45.260068    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6z6\" (UniqueName: \"kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6\") pod \"network-check-target-qcz7g\" (UID: \"6bcb360a-f63e-47c3-ae1c-eb263e42f691\") " pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:45.260301 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:45.260252    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 19:22:45.260301 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:45.260294    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 19:22:45.260390 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:45.260309    2550 projected.go:196] Error preparing data for projected volume kube-api-access-7m6z6 for pod openshift-network-diagnostics/network-check-target-qcz7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:45.260390 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:45.260375    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6 podName:6bcb360a-f63e-47c3-ae1c-eb263e42f691 nodeName:}" failed. No retries permitted until 2026-07-20 19:23:01.260357156 +0000 UTC m=+33.346308137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m6z6" (UniqueName: "kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6") pod "network-check-target-qcz7g" (UID: "6bcb360a-f63e-47c3-ae1c-eb263e42f691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:22:45.444881 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:45.444850    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:45.445323 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:45.444958    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:45.445323 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:45.445024    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:45.445323 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:45.445151    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:46.444602 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:46.444567    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:46.444755 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:46.444712    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:47.444998 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:47.444961    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:47.445432 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:47.445077    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:47.445432 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:47.445129    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:47.445432 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:47.445270    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:48.445827 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:48.445788    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:48.446314 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:48.445906    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:49.444534 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:49.444500    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:49.444705 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:49.444500    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:49.444705 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:49.444630    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:49.444826 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:49.444731    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:50.444775 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:50.444604    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:50.445193 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:50.444856    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:50.661405 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:50.661204    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-55bk8" event={"ID":"0b013675-0f9f-4383-bb33-cf662c795463","Type":"ContainerStarted","Data":"bfea8202c6636448cb9b99b3792ba2b18ed64b55241947b3407fb1ba99299832"}
Jul 20 19:22:50.663947 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:50.663923    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:22:50.664756 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:50.664689    2550 generic.go:356] "Generic (PLEG): container finished" podID="e4531c13-81bf-480d-b18e-0a31aceb78d6" containerID="ba9f1282c9addcbed2b8e1eccb233d60918087dba5ac858b24a7aafc2cc39e38" exitCode=1
Jul 20 19:22:50.664852 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:50.664757    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42skd" event={"ID":"e4531c13-81bf-480d-b18e-0a31aceb78d6","Type":"ContainerDied","Data":"ba9f1282c9addcbed2b8e1eccb233d60918087dba5ac858b24a7aafc2cc39e38"}
Jul 20 19:22:50.664923 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:50.664865    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42skd" event={"ID":"e4531c13-81bf-480d-b18e-0a31aceb78d6","Type":"ContainerStarted","Data":"d1fbdf7a0fe35d0b0ef0776f550e374ad08623ff6ab5bc8cd591a2f585d8ce35"}
Jul 20 19:22:50.666887 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:50.666829    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7xbj9" event={"ID":"d51fbd0d-1b64-4a00-9eb3-495652e656ec","Type":"ContainerStarted","Data":"0d13416c30bfb61693fd5fb306f9057caec30dd5b8b39da9eb55fd9f92571e3b"}
Jul 20 19:22:50.668664 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:50.668640    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-227.ec2.internal" event={"ID":"e2e50227d65fda4050a8a8dd57637707","Type":"ContainerStarted","Data":"059091c67708ded9b58c7eb1602f08444e150247e38e305a015dd6f299af6b94"}
Jul 20 19:22:50.679613 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:50.679547    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-55bk8" podStartSLOduration=2.330783995 podStartE2EDuration="22.679528536s" totalImagesPullingTime="20.348744541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:22:50.678526895 +0000 UTC m=+22.764477903" watchObservedRunningTime="2026-07-20 19:22:50.679528536 +0000 UTC m=+22.765479539"
Jul 20 19:22:50.711891 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:50.711847    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-227.ec2.internal" podStartSLOduration=21.711832436 podStartE2EDuration="21.711832436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:22:50.696981613 +0000 UTC m=+22.782932606" watchObservedRunningTime="2026-07-20 19:22:50.711832436 +0000 UTC m=+22.797783451"
Jul 20 19:22:50.712456 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:50.712372    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7xbj9" podStartSLOduration=2.437515054 podStartE2EDuration="22.712354162s" totalImagesPullingTime="20.274839108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:22:50.711766822 +0000 UTC m=+22.797717821" watchObservedRunningTime="2026-07-20 19:22:50.712354162 +0000 UTC m=+22.798305162"
Jul 20 19:22:51.445243 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.445211    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:51.445922 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:51.445334    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:51.445922 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.445362    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:51.445922 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:51.445408    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:51.672672 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.672639    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-54b4h" event={"ID":"d8b8de86-0507-4fb5-a218-ef90a0a8100c","Type":"ContainerStarted","Data":"15aec011585b7ad96fbc9217f27091bf21efa0ce2755503344eb7bc834a135b5"}
Jul 20 19:22:51.674050 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.674023    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2kzc8" event={"ID":"ed7bad58-74fb-44fc-9d4d-f2f04d4a1602","Type":"ContainerStarted","Data":"e483afad7b6f8bd95a6a5d8aaab0cbcc5fa106dd0d8c5f45108a153775982abd"}
Jul 20 19:22:51.675434 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.675406    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mb4tg" event={"ID":"1e7a94d4-ebe8-4991-92b5-157d5153b6ef","Type":"ContainerStarted","Data":"bc5487b68acfb270be81fe447e8110f8d831f8c9def55567c864865f03f17d15"}
Jul 20 19:22:51.676746 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.676723    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4v55" event={"ID":"fb7692a4-c508-46e6-90c6-a21fb9450d3a","Type":"ContainerStarted","Data":"e104e11b61b7f5d3c58561480c56834c18833f48b77103d0c814c3221248023c"}
Jul 20 19:22:51.678092 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.678062    2550 generic.go:356] "Generic (PLEG): container finished" podID="8c41f37e7a40aac129466553a91e6d57" containerID="c476a5de1fcba71524c0ece85032ce0de2865d776d19faa09e0c031102a4af3c" exitCode=0
Jul 20 19:22:51.678183 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.678137    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal" event={"ID":"8c41f37e7a40aac129466553a91e6d57","Type":"ContainerDied","Data":"c476a5de1fcba71524c0ece85032ce0de2865d776d19faa09e0c031102a4af3c"}
Jul 20 19:22:51.679491 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.679472    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz" event={"ID":"2be025f3-18ef-47df-a00b-d5fcd57bcc72","Type":"ContainerStarted","Data":"7e54c2beaf9ef6c92f1ebd07e346a940cb3467e0e082c0b690823bd114c23a2b"}
Jul 20 19:22:51.681922 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.681906    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:22:51.682212 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.682195    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42skd" event={"ID":"e4531c13-81bf-480d-b18e-0a31aceb78d6","Type":"ContainerStarted","Data":"f44a1606b3c26b2eb0c8d85a35f6c6a39cc59d8ee9a2ccf3043bdff66cd1fdc7"}
Jul 20 19:22:51.682287 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.682247    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42skd" event={"ID":"e4531c13-81bf-480d-b18e-0a31aceb78d6","Type":"ContainerStarted","Data":"ea26a7b8ffa6dccc20a269de376f116d4bd38c7b0c457098c880ee5b9e5676e9"}
Jul 20 19:22:51.682333 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.682311    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42skd" event={"ID":"e4531c13-81bf-480d-b18e-0a31aceb78d6","Type":"ContainerStarted","Data":"28a473453e8ffccec7c62c4232c553b16c118c5be62be72a226330df037898d5"}
Jul 20 19:22:51.682369 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.682346    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42skd" event={"ID":"e4531c13-81bf-480d-b18e-0a31aceb78d6","Type":"ContainerStarted","Data":"f0533f6526eb4b8d632854a66b62ca418ee4356177acf32333ce1fa1fec66735"}
Jul 20 19:22:51.683564 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.683545    2550 generic.go:356] "Generic (PLEG): container finished" podID="394d8ca9-d411-4acf-b1d3-bc7108992e1d" containerID="b9d26e2fc048adbeb0ec0559973de0c123942652172c58481b0bfbc165621dbd" exitCode=0
Jul 20 19:22:51.683647 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.683622    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m7fr" event={"ID":"394d8ca9-d411-4acf-b1d3-bc7108992e1d","Type":"ContainerDied","Data":"b9d26e2fc048adbeb0ec0559973de0c123942652172c58481b0bfbc165621dbd"}
Jul 20 19:22:51.698642 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.698605    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-54b4h" podStartSLOduration=3.349033998 podStartE2EDuration="23.698590302s" totalImagesPullingTime="20.349556304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:22:51.686042918 +0000 UTC m=+23.771994109" watchObservedRunningTime="2026-07-20 19:22:51.698590302 +0000 UTC m=+23.784541300"
Jul 20 19:22:51.699325 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.699278    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c4v55" podStartSLOduration=3.364664476 podStartE2EDuration="23.699251743s" totalImagesPullingTime="20.334587267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:22:51.698840484 +0000 UTC m=+23.784791486" watchObservedRunningTime="2026-07-20 19:22:51.699251743 +0000 UTC m=+23.785202762"
Jul 20 19:22:51.713413 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.713366    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mb4tg" podStartSLOduration=11.751110336 podStartE2EDuration="23.713348926s" totalImagesPullingTime="11.96223859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:22:51.713320132 +0000 UTC m=+23.799271133" watchObservedRunningTime="2026-07-20 19:22:51.713348926 +0000 UTC m=+23.799299926"
Jul 20 19:22:51.728356 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.728315    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2kzc8" podStartSLOduration=3.427587587 podStartE2EDuration="23.728301522s" totalImagesPullingTime="20.300713935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:22:51.727688255 +0000 UTC m=+23.813639255" watchObservedRunningTime="2026-07-20 19:22:51.728301522 +0000 UTC m=+23.814252520"
Jul 20 19:22:51.811797 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.811578    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:51.813014 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:51.812987    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:52.444501 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:52.444474    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:52.444634 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:52.444609    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:52.453115 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:52.453093    2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 20 19:22:52.688007 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:52.687974    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal" event={"ID":"8c41f37e7a40aac129466553a91e6d57","Type":"ContainerStarted","Data":"9e389ea582f8fdb2ed888e21df66b74957da4d8dd86fd8a67eed27d2d698f2ca"}
Jul 20 19:22:52.689745 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:52.689716    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz" event={"ID":"2be025f3-18ef-47df-a00b-d5fcd57bcc72","Type":"ContainerStarted","Data":"2210f4ed8db18107e997acce8e426fab4255ec3b8b7ec7c801771f4d373d54d8"}
Jul 20 19:22:52.690085 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:52.690069    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:52.690497 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:52.690475    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mb4tg"
Jul 20 19:22:52.702776 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:52.702736    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-227.ec2.internal" podStartSLOduration=23.70272485 podStartE2EDuration="23.70272485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:22:52.702350965 +0000 UTC m=+24.788301964" watchObservedRunningTime="2026-07-20 19:22:52.70272485 +0000 UTC m=+24.788675848"
Jul 20 19:22:53.390380 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:53.390251    2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-20T19:22:52.453110616Z","UUID":"d3ea13f9-0aa8-49c0-ba92-d89d184fd118","Handler":null,"Name":"","Endpoint":""}
Jul 20 19:22:53.392096 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:53.392066    2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 20 19:22:53.392240 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:53.392113    2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 20 19:22:53.445018 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:53.444989    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:53.445195 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:53.445120    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:53.445195 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:53.445138    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:53.445338 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:53.445216    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:53.697370 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:53.697350    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:22:53.697800 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:53.697776    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42skd" event={"ID":"e4531c13-81bf-480d-b18e-0a31aceb78d6","Type":"ContainerStarted","Data":"8c525737d6a7c66cf4294693bd6fdaea6dddffc1f4d52cbb7aedf57fe1881381"}
Jul 20 19:22:54.445119 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:54.445079    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:54.445321 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:54.445210    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:54.702156 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:54.702056    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz" event={"ID":"2be025f3-18ef-47df-a00b-d5fcd57bcc72","Type":"ContainerStarted","Data":"11122131d3d7190c0ffbeab265b0b71d68236f22609f2efd16a696356062e35e"}
Jul 20 19:22:54.719306 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:54.719248    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k4nvz" podStartSLOduration=4.183046701 podStartE2EDuration="26.719231775s" totalImagesPullingTime="22.536185074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:22:54.719117258 +0000 UTC m=+26.805068257" watchObservedRunningTime="2026-07-20 19:22:54.719231775 +0000 UTC m=+26.805182774"
Jul 20 19:22:55.445064 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:55.444898    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:55.445284 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:55.444950    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:55.445284 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:55.445159    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:55.445404 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:55.445277    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:55.708124 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:55.708094    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:22:55.708769 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:55.708612    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42skd" event={"ID":"e4531c13-81bf-480d-b18e-0a31aceb78d6","Type":"ContainerStarted","Data":"1f2f000301c4a9ad0c7ebfbde027b2b4350521df86a1d33c26800ad108ad94a1"}
Jul 20 19:22:55.708991 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:55.708972    2550 scope.go:117] "RemoveContainer" containerID="ba9f1282c9addcbed2b8e1eccb233d60918087dba5ac858b24a7aafc2cc39e38"
Jul 20 19:22:56.444570 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:56.444452    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:56.444732 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:56.444591    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:57.444891 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.444683    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:57.445376 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.444728    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:57.445376 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:57.445017    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:57.445376 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:57.445051    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:57.607280 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.607233    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ww9zr"]
Jul 20 19:22:57.607490 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.607379    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:57.607555 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:57.607485    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:57.609837 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.609806    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qcz7g"]
Jul 20 19:22:57.610366 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.610346    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6bdk5"]
Jul 20 19:22:57.716741 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.716679    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:22:57.717096 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.717075    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-42skd" event={"ID":"e4531c13-81bf-480d-b18e-0a31aceb78d6","Type":"ContainerStarted","Data":"da73e1291fabe4486e63503d7dedbfa7f37388035e7668d84a5cec49b9bdc1ce"}
Jul 20 19:22:57.717330 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.717305    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:57.717436 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.717391    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:57.717501 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.717436    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:57.719173 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.719147    2550 generic.go:356] "Generic (PLEG): container finished" podID="394d8ca9-d411-4acf-b1d3-bc7108992e1d" containerID="b53bfdb034a95ba91590615e430f00ffcf81fac08bc50656c6e7840bddb4e966" exitCode=0
Jul 20 19:22:57.719313 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.719216    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:57.719313 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.719230    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m7fr" event={"ID":"394d8ca9-d411-4acf-b1d3-bc7108992e1d","Type":"ContainerDied","Data":"b53bfdb034a95ba91590615e430f00ffcf81fac08bc50656c6e7840bddb4e966"}
Jul 20 19:22:57.719437 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:57.719342    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:57.719437 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.719383    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:57.719547 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:57.719493    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:57.733288 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.733232    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:57.733458 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.733444    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:22:57.746375 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:57.746336    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-42skd" podStartSLOduration=9.16776924 podStartE2EDuration="29.746322566s" totalImagesPullingTime="20.578553326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:22:57.745977219 +0000 UTC m=+29.831928234" watchObservedRunningTime="2026-07-20 19:22:57.746322566 +0000 UTC m=+29.832273563"
Jul 20 19:22:59.445198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:59.445156    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:22:59.445776 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:59.445165    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:22:59.445776 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:59.445292    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:22:59.445776 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:59.445296    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:22:59.445776 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:59.445362    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:22:59.445776 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:22:59.445450    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:22:59.726655 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:59.726618    2550 generic.go:356] "Generic (PLEG): container finished" podID="394d8ca9-d411-4acf-b1d3-bc7108992e1d" containerID="60dedc15a4a2ec758cf817d8a552dcf6c754a4dd9de62f3ed70e219ac165b10c" exitCode=0
Jul 20 19:22:59.726814 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:22:59.726685    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m7fr" event={"ID":"394d8ca9-d411-4acf-b1d3-bc7108992e1d","Type":"ContainerDied","Data":"60dedc15a4a2ec758cf817d8a552dcf6c754a4dd9de62f3ed70e219ac165b10c"}
Jul 20 19:23:01.080031 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:01.079954    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:23:01.080387 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:01.080075    2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 20 19:23:01.080387 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:01.080125    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret podName:57db0719-7a2e-49bf-9356-8b1ea7b94069 nodeName:}" failed. No retries permitted until 2026-07-20 19:23:33.080112463 +0000 UTC m=+65.166063438 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret") pod "global-pull-secret-syncer-ww9zr" (UID: "57db0719-7a2e-49bf-9356-8b1ea7b94069") : object "kube-system"/"original-pull-secret" not registered
Jul 20 19:23:01.181338 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:01.181299    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:23:01.181507 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:01.181439    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:23:01.181507 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:01.181502    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs podName:cd899fed-afcf-484c-89bf-28566c07b2d2 nodeName:}" failed. No retries permitted until 2026-07-20 19:23:33.181486837 +0000 UTC m=+65.267437812 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs") pod "network-metrics-daemon-6bdk5" (UID: "cd899fed-afcf-484c-89bf-28566c07b2d2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 20 19:23:01.282025 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:01.281986    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6z6\" (UniqueName: \"kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6\") pod \"network-check-target-qcz7g\" (UID: \"6bcb360a-f63e-47c3-ae1c-eb263e42f691\") " pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:23:01.282186 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:01.282128    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 20 19:23:01.282186 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:01.282150    2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 20 19:23:01.282186 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:01.282161    2550 projected.go:196] Error preparing data for projected volume kube-api-access-7m6z6 for pod openshift-network-diagnostics/network-check-target-qcz7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:23:01.282302 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:01.282209    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6 podName:6bcb360a-f63e-47c3-ae1c-eb263e42f691 nodeName:}" failed. No retries permitted until 2026-07-20 19:23:33.282195473 +0000 UTC m=+65.368146449 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7m6z6" (UniqueName: "kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6") pod "network-check-target-qcz7g" (UID: "6bcb360a-f63e-47c3-ae1c-eb263e42f691") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 20 19:23:01.445135 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:01.445094    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:23:01.445135 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:01.445121    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:23:01.445394 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:01.445113    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:23:01.445394 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:01.445273    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:23:01.445394 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:01.445310    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ww9zr" podUID="57db0719-7a2e-49bf-9356-8b1ea7b94069"
Jul 20 19:23:01.445394 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:01.445378    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qcz7g" podUID="6bcb360a-f63e-47c3-ae1c-eb263e42f691"
Jul 20 19:23:01.732738 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:01.732650    2550 generic.go:356] "Generic (PLEG): container finished" podID="394d8ca9-d411-4acf-b1d3-bc7108992e1d" containerID="b4d4ac4409d9d61b7a6a6f484dc4f7b70eeb11d4b731d4182b47f01d83a16cd0" exitCode=0
Jul 20 19:23:01.732738 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:01.732694    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m7fr" event={"ID":"394d8ca9-d411-4acf-b1d3-bc7108992e1d","Type":"ContainerDied","Data":"b4d4ac4409d9d61b7a6a6f484dc4f7b70eeb11d4b731d4182b47f01d83a16cd0"}
Jul 20 19:23:03.147384 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.147361    2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-227.ec2.internal" event="NodeReady"
Jul 20 19:23:03.147785 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.147474    2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 20 19:23:03.209339 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.209305    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vlxp5"]
Jul 20 19:23:03.231696 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.231659    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gk5f7"]
Jul 20 19:23:03.231871 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.231803    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.234204 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.233935    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 20 19:23:03.234204 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.233952    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 20 19:23:03.234204 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.233963    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bsgbb\""
Jul 20 19:23:03.247514 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.247478    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vlxp5"]
Jul 20 19:23:03.247634 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.247516    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gk5f7"
Jul 20 19:23:03.247634 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.247548    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gk5f7"]
Jul 20 19:23:03.249781 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.249759    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 20 19:23:03.249891 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.249784    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 20 19:23:03.250108 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.250063    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-46bg7\""
Jul 20 19:23:03.250455 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.250122    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 20 19:23:03.398959 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.398887    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fe76da7-11be-40ce-81e9-8e57306a184d-config-volume\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.398959 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.398917    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe76da7-11be-40ce-81e9-8e57306a184d-tmp-dir\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.398959 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.398936    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnpvs\" (UniqueName: \"kubernetes.io/projected/471d7512-1535-4263-95db-a1fb1e7bd134-kube-api-access-rnpvs\") pod \"ingress-canary-gk5f7\" (UID: \"471d7512-1535-4263-95db-a1fb1e7bd134\") " pod="openshift-ingress-canary/ingress-canary-gk5f7"
Jul 20 19:23:03.398959 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.398955    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.399236 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.398975    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/471d7512-1535-4263-95db-a1fb1e7bd134-cert\") pod \"ingress-canary-gk5f7\" (UID: \"471d7512-1535-4263-95db-a1fb1e7bd134\") " pod="openshift-ingress-canary/ingress-canary-gk5f7"
Jul 20 19:23:03.399236 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.399097    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqt7\" (UniqueName: \"kubernetes.io/projected/2fe76da7-11be-40ce-81e9-8e57306a184d-kube-api-access-dkqt7\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.445197 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.445161    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:23:03.445372 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.445292    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:23:03.445565 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.445533    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:23:03.448024 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.447728    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 19:23:03.448024 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.447722    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gzsbm\""
Jul 20 19:23:03.448466 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.448447    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 19:23:03.448584 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.448559    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 19:23:03.448768 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.448748    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 19:23:03.451342 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.449139    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6ctrm\""
Jul 20 19:23:03.500448 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.500411    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqt7\" (UniqueName: \"kubernetes.io/projected/2fe76da7-11be-40ce-81e9-8e57306a184d-kube-api-access-dkqt7\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.500601 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.500503    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fe76da7-11be-40ce-81e9-8e57306a184d-config-volume\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.500601 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.500537    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe76da7-11be-40ce-81e9-8e57306a184d-tmp-dir\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.500601 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.500562    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnpvs\" (UniqueName: \"kubernetes.io/projected/471d7512-1535-4263-95db-a1fb1e7bd134-kube-api-access-rnpvs\") pod \"ingress-canary-gk5f7\" (UID: \"471d7512-1535-4263-95db-a1fb1e7bd134\") " pod="openshift-ingress-canary/ingress-canary-gk5f7"
Jul 20 19:23:03.500601 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.500592    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.500800 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.500623    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/471d7512-1535-4263-95db-a1fb1e7bd134-cert\") pod \"ingress-canary-gk5f7\" (UID: \"471d7512-1535-4263-95db-a1fb1e7bd134\") " pod="openshift-ingress-canary/ingress-canary-gk5f7"
Jul 20 19:23:03.500800 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:03.500788    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 19:23:03.500881 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:03.500859    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls podName:2fe76da7-11be-40ce-81e9-8e57306a184d nodeName:}" failed. No retries permitted until 2026-07-20 19:23:04.000839376 +0000 UTC m=+36.086790361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls") pod "dns-default-vlxp5" (UID: "2fe76da7-11be-40ce-81e9-8e57306a184d") : secret "dns-default-metrics-tls" not found
Jul 20 19:23:03.501179 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.501153    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fe76da7-11be-40ce-81e9-8e57306a184d-tmp-dir\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.501626 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.501433    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2fe76da7-11be-40ce-81e9-8e57306a184d-config-volume\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.505300 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.505278    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/471d7512-1535-4263-95db-a1fb1e7bd134-cert\") pod \"ingress-canary-gk5f7\" (UID: \"471d7512-1535-4263-95db-a1fb1e7bd134\") " pod="openshift-ingress-canary/ingress-canary-gk5f7"
Jul 20 19:23:03.510837 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.510811    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnpvs\" (UniqueName: \"kubernetes.io/projected/471d7512-1535-4263-95db-a1fb1e7bd134-kube-api-access-rnpvs\") pod \"ingress-canary-gk5f7\" (UID: \"471d7512-1535-4263-95db-a1fb1e7bd134\") " pod="openshift-ingress-canary/ingress-canary-gk5f7"
Jul 20 19:23:03.511402 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.511377    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqt7\" (UniqueName: \"kubernetes.io/projected/2fe76da7-11be-40ce-81e9-8e57306a184d-kube-api-access-dkqt7\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:03.557359 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.557328    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gk5f7"
Jul 20 19:23:03.780527 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:03.780499    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gk5f7"]
Jul 20 19:23:03.784449 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:23:03.784421    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471d7512_1535_4263_95db_a1fb1e7bd134.slice/crio-df90fbc16f279d04f980eb4951ee1233f1b6ac25505ea01f6d5193b6bbfa646f WatchSource:0}: Error finding container df90fbc16f279d04f980eb4951ee1233f1b6ac25505ea01f6d5193b6bbfa646f: Status 404 returned error can't find the container with id df90fbc16f279d04f980eb4951ee1233f1b6ac25505ea01f6d5193b6bbfa646f
Jul 20 19:23:04.003890 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:04.003801    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:04.004023 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:04.003960    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 19:23:04.004023 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:04.004024    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls podName:2fe76da7-11be-40ce-81e9-8e57306a184d nodeName:}" failed. No retries permitted until 2026-07-20 19:23:05.004006056 +0000 UTC m=+37.089957033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls") pod "dns-default-vlxp5" (UID: "2fe76da7-11be-40ce-81e9-8e57306a184d") : secret "dns-default-metrics-tls" not found
Jul 20 19:23:04.741737 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:04.741673    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gk5f7" event={"ID":"471d7512-1535-4263-95db-a1fb1e7bd134","Type":"ContainerStarted","Data":"df90fbc16f279d04f980eb4951ee1233f1b6ac25505ea01f6d5193b6bbfa646f"}
Jul 20 19:23:05.012638 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:05.012553    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:05.012761 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:05.012710    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 19:23:05.012805 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:05.012789    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls podName:2fe76da7-11be-40ce-81e9-8e57306a184d nodeName:}" failed. No retries permitted until 2026-07-20 19:23:07.012773849 +0000 UTC m=+39.098724830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls") pod "dns-default-vlxp5" (UID: "2fe76da7-11be-40ce-81e9-8e57306a184d") : secret "dns-default-metrics-tls" not found
Jul 20 19:23:07.028423 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:07.028383    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:07.028869 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:07.028502    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 19:23:07.028869 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:07.028574    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls podName:2fe76da7-11be-40ce-81e9-8e57306a184d nodeName:}" failed. No retries permitted until 2026-07-20 19:23:11.028554102 +0000 UTC m=+43.114505098 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls") pod "dns-default-vlxp5" (UID: "2fe76da7-11be-40ce-81e9-8e57306a184d") : secret "dns-default-metrics-tls" not found
Jul 20 19:23:07.751879 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:07.751621    2550 generic.go:356] "Generic (PLEG): container finished" podID="394d8ca9-d411-4acf-b1d3-bc7108992e1d" containerID="4a731da89c54c34f12bb2d7eb42572a934ec80d11fef83bbf79fc54f8db83909" exitCode=0
Jul 20 19:23:07.751879 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:07.751698    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m7fr" event={"ID":"394d8ca9-d411-4acf-b1d3-bc7108992e1d","Type":"ContainerDied","Data":"4a731da89c54c34f12bb2d7eb42572a934ec80d11fef83bbf79fc54f8db83909"}
Jul 20 19:23:07.753480 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:07.753458    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gk5f7" event={"ID":"471d7512-1535-4263-95db-a1fb1e7bd134","Type":"ContainerStarted","Data":"23ba79d4d9205fd36a81c5226150481072099027f50f526e25e0e0354f997c18"}
Jul 20 19:23:07.791901 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:07.791843    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gk5f7" podStartSLOduration=1.091862856 podStartE2EDuration="4.79182309s" totalImagesPullingTime="3.699960234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:23:07.790446674 +0000 UTC m=+39.876397672" watchObservedRunningTime="2026-07-20 19:23:07.79182309 +0000 UTC m=+39.877774089"
Jul 20 19:23:08.758670 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:08.758636    2550 generic.go:356] "Generic (PLEG): container finished" podID="394d8ca9-d411-4acf-b1d3-bc7108992e1d" containerID="89b76f85099d4eb41b2027cfdd444f9237e5500d083013b0dc826a4d17ff6a94" exitCode=0
Jul 20 19:23:08.759041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:08.758711    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m7fr" event={"ID":"394d8ca9-d411-4acf-b1d3-bc7108992e1d","Type":"ContainerDied","Data":"89b76f85099d4eb41b2027cfdd444f9237e5500d083013b0dc826a4d17ff6a94"}
Jul 20 19:23:09.764468 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:09.764436    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8m7fr" event={"ID":"394d8ca9-d411-4acf-b1d3-bc7108992e1d","Type":"ContainerStarted","Data":"d9b065d5e05469e94eb8dbcc0de4595dc939694a150df40f6d8f15439778ec71"}
Jul 20 19:23:09.789849 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:09.789733    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8m7fr" podStartSLOduration=8.165233972 podStartE2EDuration="41.7897198s" totalImagesPullingTime="33.624485828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:23:09.788156469 +0000 UTC m=+41.874107467" watchObservedRunningTime="2026-07-20 19:23:09.7897198 +0000 UTC m=+41.875670798"
Jul 20 19:23:11.055157 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:11.055117    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:11.055542 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:11.055239    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 19:23:11.055542 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:11.055316    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls podName:2fe76da7-11be-40ce-81e9-8e57306a184d nodeName:}" failed. No retries permitted until 2026-07-20 19:23:19.055298212 +0000 UTC m=+51.141249190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls") pod "dns-default-vlxp5" (UID: "2fe76da7-11be-40ce-81e9-8e57306a184d") : secret "dns-default-metrics-tls" not found
Jul 20 19:23:19.109567 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:19.109515    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:19.110104 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:19.109641    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 19:23:19.110104 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:19.109711    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls podName:2fe76da7-11be-40ce-81e9-8e57306a184d nodeName:}" failed. No retries permitted until 2026-07-20 19:23:35.109698575 +0000 UTC m=+67.195649550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls") pod "dns-default-vlxp5" (UID: "2fe76da7-11be-40ce-81e9-8e57306a184d") : secret "dns-default-metrics-tls" not found
Jul 20 19:23:29.736942 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:29.736909    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-42skd"
Jul 20 19:23:32.134290 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.134181    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"]
Jul 20 19:23:32.137475 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.137459    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:32.140218 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.140194    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 20 19:23:32.140218 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.140215    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 20 19:23:32.140510 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.140495    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 20 19:23:32.140564 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.140537    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 20 19:23:32.147859 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.147838    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"]
Jul 20 19:23:32.149907 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.149889    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"]
Jul 20 19:23:32.150015 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.149996    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.152448 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.152428    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 20 19:23:32.152547 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.152533    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 20 19:23:32.152613 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.152562    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 20 19:23:32.152613 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.152567    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 20 19:23:32.164206 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.164173    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"]
Jul 20 19:23:32.192511 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.192476    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.192653 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.192524    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/74c2ad87-a60e-4fd1-8746-e6eae85282a3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c687c9986-6mw7b\" (UID: \"74c2ad87-a60e-4fd1-8746-e6eae85282a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:32.192653 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.192583    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1b980a-5805-43aa-8e24-82433b1e186b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.192653 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.192628    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-hub\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.192773 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.192715    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.192773 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.192743    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmprf\" (UniqueName: \"kubernetes.io/projected/bf1b980a-5805-43aa-8e24-82433b1e186b-kube-api-access-wmprf\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.192773 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.192766    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdhg\" (UniqueName: \"kubernetes.io/projected/74c2ad87-a60e-4fd1-8746-e6eae85282a3-kube-api-access-cgdhg\") pod \"klusterlet-addon-workmgr-5c687c9986-6mw7b\" (UID: \"74c2ad87-a60e-4fd1-8746-e6eae85282a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:32.192882 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.192800    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-ca\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.192882 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.192815    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74c2ad87-a60e-4fd1-8746-e6eae85282a3-tmp\") pod \"klusterlet-addon-workmgr-5c687c9986-6mw7b\" (UID: \"74c2ad87-a60e-4fd1-8746-e6eae85282a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:32.293602 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.293563    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1b980a-5805-43aa-8e24-82433b1e186b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.293772 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.293631    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-hub\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.293772 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.293667    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.293772 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.293698    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmprf\" (UniqueName: \"kubernetes.io/projected/bf1b980a-5805-43aa-8e24-82433b1e186b-kube-api-access-wmprf\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.293772 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.293721    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdhg\" (UniqueName: \"kubernetes.io/projected/74c2ad87-a60e-4fd1-8746-e6eae85282a3-kube-api-access-cgdhg\") pod \"klusterlet-addon-workmgr-5c687c9986-6mw7b\" (UID: \"74c2ad87-a60e-4fd1-8746-e6eae85282a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:32.293772 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.293764    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-ca\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.294004 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.293786    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74c2ad87-a60e-4fd1-8746-e6eae85282a3-tmp\") pod \"klusterlet-addon-workmgr-5c687c9986-6mw7b\" (UID: \"74c2ad87-a60e-4fd1-8746-e6eae85282a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:32.294004 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.293817    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.294101 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.294074    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/74c2ad87-a60e-4fd1-8746-e6eae85282a3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c687c9986-6mw7b\" (UID: \"74c2ad87-a60e-4fd1-8746-e6eae85282a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:32.294192 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.294166    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74c2ad87-a60e-4fd1-8746-e6eae85282a3-tmp\") pod \"klusterlet-addon-workmgr-5c687c9986-6mw7b\" (UID: \"74c2ad87-a60e-4fd1-8746-e6eae85282a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:32.294425 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.294398    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf1b980a-5805-43aa-8e24-82433b1e186b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.297513 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.297485    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.297597 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.297515    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-hub\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.297597 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.297495    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/74c2ad87-a60e-4fd1-8746-e6eae85282a3-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c687c9986-6mw7b\" (UID: \"74c2ad87-a60e-4fd1-8746-e6eae85282a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:32.297669 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.297651    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.301569 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.301541    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmprf\" (UniqueName: \"kubernetes.io/projected/bf1b980a-5805-43aa-8e24-82433b1e186b-kube-api-access-wmprf\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.301652 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.301573    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdhg\" (UniqueName: \"kubernetes.io/projected/74c2ad87-a60e-4fd1-8746-e6eae85282a3-kube-api-access-cgdhg\") pod \"klusterlet-addon-workmgr-5c687c9986-6mw7b\" (UID: \"74c2ad87-a60e-4fd1-8746-e6eae85282a3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:32.304508 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.304491    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bf1b980a-5805-43aa-8e24-82433b1e186b-ca\") pod \"cluster-proxy-proxy-agent-7b875bffcb-tt7bd\" (UID: \"bf1b980a-5805-43aa-8e24-82433b1e186b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.446320 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.446234    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:32.471103 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.471078    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:23:32.590436 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.590406    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"]
Jul 20 19:23:32.593653 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:23:32.593620    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c2ad87_a60e_4fd1_8746_e6eae85282a3.slice/crio-9994f8d1b56d1169e40d838546e0711049fc4a6b93d9053f884846c0e60f8297 WatchSource:0}: Error finding container 9994f8d1b56d1169e40d838546e0711049fc4a6b93d9053f884846c0e60f8297: Status 404 returned error can't find the container with id 9994f8d1b56d1169e40d838546e0711049fc4a6b93d9053f884846c0e60f8297
Jul 20 19:23:32.620906 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.620876    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"]
Jul 20 19:23:32.623653 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:23:32.623632    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf1b980a_5805_43aa_8e24_82433b1e186b.slice/crio-b535516ae70dab44ab83676abdd6227793891171cd5d3b2a5ad87ca499c90e4d WatchSource:0}: Error finding container b535516ae70dab44ab83676abdd6227793891171cd5d3b2a5ad87ca499c90e4d: Status 404 returned error can't find the container with id b535516ae70dab44ab83676abdd6227793891171cd5d3b2a5ad87ca499c90e4d
Jul 20 19:23:32.817600 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.817507    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b" event={"ID":"74c2ad87-a60e-4fd1-8746-e6eae85282a3","Type":"ContainerStarted","Data":"9994f8d1b56d1169e40d838546e0711049fc4a6b93d9053f884846c0e60f8297"}
Jul 20 19:23:32.818577 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:32.818555    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" event={"ID":"bf1b980a-5805-43aa-8e24-82433b1e186b","Type":"ContainerStarted","Data":"b535516ae70dab44ab83676abdd6227793891171cd5d3b2a5ad87ca499c90e4d"}
Jul 20 19:23:33.100789 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.100699    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:23:33.103126 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.103109    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 20 19:23:33.113274 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.113238    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/57db0719-7a2e-49bf-9356-8b1ea7b94069-original-pull-secret\") pod \"global-pull-secret-syncer-ww9zr\" (UID: \"57db0719-7a2e-49bf-9356-8b1ea7b94069\") " pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:23:33.166804 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.166767    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ww9zr"
Jul 20 19:23:33.202108 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.202072    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:23:33.204674 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.204650    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 20 19:23:33.213203 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:33.213164    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 20 19:23:33.213436 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:33.213352    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs podName:cd899fed-afcf-484c-89bf-28566c07b2d2 nodeName:}" failed. No retries permitted until 2026-07-20 19:24:37.213245232 +0000 UTC m=+129.299196230 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs") pod "network-metrics-daemon-6bdk5" (UID: "cd899fed-afcf-484c-89bf-28566c07b2d2") : secret "metrics-daemon-secret" not found
Jul 20 19:23:33.303739 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.303312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6z6\" (UniqueName: \"kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6\") pod \"network-check-target-qcz7g\" (UID: \"6bcb360a-f63e-47c3-ae1c-eb263e42f691\") " pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:23:33.306436 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.306088    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 20 19:23:33.316546 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.316517    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 20 19:23:33.327989 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.327948    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m6z6\" (UniqueName: \"kubernetes.io/projected/6bcb360a-f63e-47c3-ae1c-eb263e42f691-kube-api-access-7m6z6\") pod \"network-check-target-qcz7g\" (UID: \"6bcb360a-f63e-47c3-ae1c-eb263e42f691\") " pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:23:33.335430 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.334754    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ww9zr"]
Jul 20 19:23:33.341050 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:23:33.341018    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57db0719_7a2e_49bf_9356_8b1ea7b94069.slice/crio-4ca8d30a23817045f03f80b8402ede0cf9966050f510bb59b7159cc461c99a72 WatchSource:0}: Error finding container 4ca8d30a23817045f03f80b8402ede0cf9966050f510bb59b7159cc461c99a72: Status 404 returned error can't find the container with id 4ca8d30a23817045f03f80b8402ede0cf9966050f510bb59b7159cc461c99a72
Jul 20 19:23:33.462397 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.462070    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6ctrm\""
Jul 20 19:23:33.470585 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.470558    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:23:33.655865 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.655832    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qcz7g"]
Jul 20 19:23:33.661046 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:23:33.661018    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bcb360a_f63e_47c3_ae1c_eb263e42f691.slice/crio-6332c1872a67e9a4505e8736da894bd9f1b576c5682e3e0247375521f3a1bb01 WatchSource:0}: Error finding container 6332c1872a67e9a4505e8736da894bd9f1b576c5682e3e0247375521f3a1bb01: Status 404 returned error can't find the container with id 6332c1872a67e9a4505e8736da894bd9f1b576c5682e3e0247375521f3a1bb01
Jul 20 19:23:33.829916 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.829877    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ww9zr" event={"ID":"57db0719-7a2e-49bf-9356-8b1ea7b94069","Type":"ContainerStarted","Data":"4ca8d30a23817045f03f80b8402ede0cf9966050f510bb59b7159cc461c99a72"}
Jul 20 19:23:33.832836 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:33.832809    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qcz7g" event={"ID":"6bcb360a-f63e-47c3-ae1c-eb263e42f691","Type":"ContainerStarted","Data":"6332c1872a67e9a4505e8736da894bd9f1b576c5682e3e0247375521f3a1bb01"}
Jul 20 19:23:35.119672 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:35.119631    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:23:35.120124 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:35.119790    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 19:23:35.120124 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:23:35.119882    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls podName:2fe76da7-11be-40ce-81e9-8e57306a184d nodeName:}" failed. No retries permitted until 2026-07-20 19:24:07.119861513 +0000 UTC m=+99.205812502 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls") pod "dns-default-vlxp5" (UID: "2fe76da7-11be-40ce-81e9-8e57306a184d") : secret "dns-default-metrics-tls" not found
Jul 20 19:23:41.860881 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.860845    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ww9zr" event={"ID":"57db0719-7a2e-49bf-9356-8b1ea7b94069","Type":"ContainerStarted","Data":"c3294ea1122b7ba94d4e5661e17acc9aa0277b1c65e96f4efc87a1b563fd8b59"}
Jul 20 19:23:41.862251 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.862232    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b" event={"ID":"74c2ad87-a60e-4fd1-8746-e6eae85282a3","Type":"ContainerStarted","Data":"fc68975e58575ee2147af61fff823d7c06f8d65e26f6b3ce775b339d099b5545"}
Jul 20 19:23:41.862443 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.862428    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:41.863710 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.863675    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qcz7g" event={"ID":"6bcb360a-f63e-47c3-ae1c-eb263e42f691","Type":"ContainerStarted","Data":"5261d36409b42242e2c9f1a6f52b08cb9d1c8a12ab553572c50fea7956302a6f"}
Jul 20 19:23:41.863845 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.863740    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:23:41.864306 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.864287    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:23:41.865584 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.865566    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" event={"ID":"bf1b980a-5805-43aa-8e24-82433b1e186b","Type":"ContainerStarted","Data":"640ead496d2d752660be7a8d83d6a990166a18c55fa1da8038b446d6cae16c49"}
Jul 20 19:23:41.865665 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.865591    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" event={"ID":"bf1b980a-5805-43aa-8e24-82433b1e186b","Type":"ContainerStarted","Data":"6035bea2fcd3a3a02b1704f60904d5ba35d78790895a1149cc6660660784da1f"}
Jul 20 19:23:41.865665 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.865603    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" event={"ID":"bf1b980a-5805-43aa-8e24-82433b1e186b","Type":"ContainerStarted","Data":"f36b99cccd64bdaef538ff1b3403d024bc6b2e3ee554d7f1649d82a0bb8fa427"}
Jul 20 19:23:41.877517 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.877482    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ww9zr" podStartSLOduration=66.06551081 podStartE2EDuration="1m13.877459084s" totalImagesPullingTime="7.811948274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:23:41.877288271 +0000 UTC m=+73.963239267" watchObservedRunningTime="2026-07-20 19:23:41.877459084 +0000 UTC m=+73.963410074"
Jul 20 19:23:41.891409 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.891358    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qcz7g" podStartSLOduration=66.400704289 podStartE2EDuration="1m13.891344989s" totalImagesPullingTime="7.4906407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:23:41.890457409 +0000 UTC m=+73.976408405" watchObservedRunningTime="2026-07-20 19:23:41.891344989 +0000 UTC m=+73.977295993"
Jul 20 19:23:41.907341 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.907296    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" podStartSLOduration=1.3860559430000001 podStartE2EDuration="9.907283465s" totalImagesPullingTime="8.521227522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:23:41.906520776 +0000 UTC m=+73.992471772" watchObservedRunningTime="2026-07-20 19:23:41.907283465 +0000 UTC m=+73.993234453"
Jul 20 19:23:41.919880 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:23:41.919843    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b" podStartSLOduration=1.356379635 podStartE2EDuration="9.919831603s" totalImagesPullingTime="8.563451968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:23:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:23:41.919675633 +0000 UTC m=+74.005626632" watchObservedRunningTime="2026-07-20 19:23:41.919831603 +0000 UTC m=+74.005782634"
Jul 20 19:24:07.149037 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:24:07.148986    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:24:07.149481 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:24:07.149132    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 19:24:07.149481 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:24:07.149200    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls podName:2fe76da7-11be-40ce-81e9-8e57306a184d nodeName:}" failed. No retries permitted until 2026-07-20 19:25:11.149181235 +0000 UTC m=+163.235132210 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls") pod "dns-default-vlxp5" (UID: "2fe76da7-11be-40ce-81e9-8e57306a184d") : secret "dns-default-metrics-tls" not found
Jul 20 19:24:12.871692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:24:12.871662    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qcz7g"
Jul 20 19:24:37.256368 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:24:37.256310    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:24:37.256852 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:24:37.256456    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 20 19:24:37.256852 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:24:37.256516    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs podName:cd899fed-afcf-484c-89bf-28566c07b2d2 nodeName:}" failed. No retries permitted until 2026-07-20 19:26:39.256500626 +0000 UTC m=+251.342451606 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs") pod "network-metrics-daemon-6bdk5" (UID: "cd899fed-afcf-484c-89bf-28566c07b2d2") : secret "metrics-daemon-secret" not found
Jul 20 19:25:06.244389 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:25:06.244345    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vlxp5" podUID="2fe76da7-11be-40ce-81e9-8e57306a184d"
Jul 20 19:25:06.472603 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:25:06.472551    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:25:07.104101 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:25:07.104073    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:25:11.186848 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:25:11.186785    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:25:11.187294 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:25:11.186956    2550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 20 19:25:11.187294 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:25:11.187033    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls podName:2fe76da7-11be-40ce-81e9-8e57306a184d nodeName:}" failed. No retries permitted until 2026-07-20 19:27:13.18701735 +0000 UTC m=+285.272968327 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls") pod "dns-default-vlxp5" (UID: "2fe76da7-11be-40ce-81e9-8e57306a184d") : secret "dns-default-metrics-tls" not found
Jul 20 19:25:17.445052 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:25:17.445017    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:25:41.863435 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:25:41.863383    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b" podUID="74c2ad87-a60e-4fd1-8746-e6eae85282a3" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
Jul 20 19:25:42.198949 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:25:42.198915    2550 generic.go:356] "Generic (PLEG): container finished" podID="74c2ad87-a60e-4fd1-8746-e6eae85282a3" containerID="fc68975e58575ee2147af61fff823d7c06f8d65e26f6b3ce775b339d099b5545" exitCode=1
Jul 20 19:25:42.199144 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:25:42.198959    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b" event={"ID":"74c2ad87-a60e-4fd1-8746-e6eae85282a3","Type":"ContainerDied","Data":"fc68975e58575ee2147af61fff823d7c06f8d65e26f6b3ce775b339d099b5545"}
Jul 20 19:25:42.199344 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:25:42.199326    2550 scope.go:117] "RemoveContainer" containerID="fc68975e58575ee2147af61fff823d7c06f8d65e26f6b3ce775b339d099b5545"
Jul 20 19:25:42.448146 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:25:42.448119    2550 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:25:43.203962 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:25:43.203925    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b" event={"ID":"74c2ad87-a60e-4fd1-8746-e6eae85282a3","Type":"ContainerStarted","Data":"d31cb8a5a7afa07fa8718713d65022834c0a2b6e31911d7b17c7d6fb040f3009"}
Jul 20 19:25:43.204390 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:25:43.204104    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:25:43.205021 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:25:43.205001    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c687c9986-6mw7b"
Jul 20 19:26:36.405338 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.405298    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7lfs6"]
Jul 20 19:26:36.407473 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.407453    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.409643 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.409624    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 20 19:26:36.410249 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.410234    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 20 19:26:36.410472 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.410451    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 20 19:26:36.410586 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.410511    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6cgzv\""
Jul 20 19:26:36.411057 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.411040    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 20 19:26:36.426293 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.426250    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7lfs6"]
Jul 20 19:26:36.568450 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.568409    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.568630 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.568470    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tql92\" (UniqueName: \"kubernetes.io/projected/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-kube-api-access-tql92\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.568630 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.568570    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-crio-socket\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.568630 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.568616    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-data-volume\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.568755 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.568640    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.669729 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.669636    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tql92\" (UniqueName: \"kubernetes.io/projected/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-kube-api-access-tql92\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.669729 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.669704    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-crio-socket\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.669729 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.669736    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-data-volume\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.669998 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.669751    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.669998 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.669771    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.669998 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.669827    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-crio-socket\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.669998 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:36.669852    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 19:26:36.669998 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:36.669860    2550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 20 19:26:36.669998 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:36.669904    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls podName:deddaa5c-cfd1-42bf-99e5-4a1f3b434a11 nodeName:}" failed. No retries permitted until 2026-07-20 19:26:37.169890084 +0000 UTC m=+249.255841060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7lfs6" (UID: "deddaa5c-cfd1-42bf-99e5-4a1f3b434a11") : secret "insights-runtime-extractor-tls" not found
Jul 20 19:26:36.669998 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:36.669935    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-kube-rbac-proxy-cm podName:deddaa5c-cfd1-42bf-99e5-4a1f3b434a11 nodeName:}" failed. No retries permitted until 2026-07-20 19:26:37.169917029 +0000 UTC m=+249.255868029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7lfs6" (UID: "deddaa5c-cfd1-42bf-99e5-4a1f3b434a11") : configmap "kube-rbac-proxy" not found
Jul 20 19:26:36.670228 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.670097    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-data-volume\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:36.679490 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:36.679460    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tql92\" (UniqueName: \"kubernetes.io/projected/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-kube-api-access-tql92\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:37.173598 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:37.173561    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:37.173790 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:37.173668    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:37.173790 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:37.173724    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 19:26:37.173904 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:37.173798    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls podName:deddaa5c-cfd1-42bf-99e5-4a1f3b434a11 nodeName:}" failed. No retries permitted until 2026-07-20 19:26:38.173781355 +0000 UTC m=+250.259732347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7lfs6" (UID: "deddaa5c-cfd1-42bf-99e5-4a1f3b434a11") : secret "insights-runtime-extractor-tls" not found
Jul 20 19:26:37.174197 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:37.174176    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:38.171526 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:38.171494    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2kzc8_ed7bad58-74fb-44fc-9d4d-f2f04d4a1602/dns-node-resolver/0.log"
Jul 20 19:26:38.181404 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:38.181374    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:38.181528 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:38.181511    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 19:26:38.181570 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:38.181566    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls podName:deddaa5c-cfd1-42bf-99e5-4a1f3b434a11 nodeName:}" failed. No retries permitted until 2026-07-20 19:26:40.181551212 +0000 UTC m=+252.267502188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7lfs6" (UID: "deddaa5c-cfd1-42bf-99e5-4a1f3b434a11") : secret "insights-runtime-extractor-tls" not found
Jul 20 19:26:39.173298 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:39.173250    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-54b4h_d8b8de86-0507-4fb5-a218-ef90a0a8100c/node-ca/0.log"
Jul 20 19:26:39.290648 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:39.290612    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:26:39.290835 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:39.290744    2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 20 19:26:39.290835 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:39.290808    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs podName:cd899fed-afcf-484c-89bf-28566c07b2d2 nodeName:}" failed. No retries permitted until 2026-07-20 19:28:41.29079291 +0000 UTC m=+373.376743886 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs") pod "network-metrics-daemon-6bdk5" (UID: "cd899fed-afcf-484c-89bf-28566c07b2d2") : secret "metrics-daemon-secret" not found
Jul 20 19:26:40.172100 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:40.172070    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gk5f7_471d7512-1535-4263-95db-a1fb1e7bd134/serve-healthcheck-canary/0.log"
Jul 20 19:26:40.197308 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:40.197278    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:40.197745 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:40.197436    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 19:26:40.197745 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:40.197529    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls podName:deddaa5c-cfd1-42bf-99e5-4a1f3b434a11 nodeName:}" failed. No retries permitted until 2026-07-20 19:26:44.197502326 +0000 UTC m=+256.283453317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7lfs6" (UID: "deddaa5c-cfd1-42bf-99e5-4a1f3b434a11") : secret "insights-runtime-extractor-tls" not found
Jul 20 19:26:44.227378 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:44.227338    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:44.227833 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:44.227447    2550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 20 19:26:44.227833 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:26:44.227497    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls podName:deddaa5c-cfd1-42bf-99e5-4a1f3b434a11 nodeName:}" failed. No retries permitted until 2026-07-20 19:26:52.227484686 +0000 UTC m=+264.313435662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7lfs6" (UID: "deddaa5c-cfd1-42bf-99e5-4a1f3b434a11") : secret "insights-runtime-extractor-tls" not found
Jul 20 19:26:52.288271 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:52.288177    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:52.290704 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:52.290675    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/deddaa5c-cfd1-42bf-99e5-4a1f3b434a11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7lfs6\" (UID: \"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11\") " pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:52.315774 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:52.315753    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7lfs6"
Jul 20 19:26:52.442156 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:52.442118    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7lfs6"]
Jul 20 19:26:52.446008 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:26:52.445980    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeddaa5c_cfd1_42bf_99e5_4a1f3b434a11.slice/crio-fed8d042eaf39a085f8d6b097fd56f31874b1a49f70e70f461ecd4814a8269fc WatchSource:0}: Error finding container fed8d042eaf39a085f8d6b097fd56f31874b1a49f70e70f461ecd4814a8269fc: Status 404 returned error can't find the container with id fed8d042eaf39a085f8d6b097fd56f31874b1a49f70e70f461ecd4814a8269fc
Jul 20 19:26:53.404097 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:53.404056    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7lfs6" event={"ID":"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11","Type":"ContainerStarted","Data":"fed8d042eaf39a085f8d6b097fd56f31874b1a49f70e70f461ecd4814a8269fc"}
Jul 20 19:26:57.418928 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:26:57.418835    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7lfs6" event={"ID":"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11","Type":"ContainerStarted","Data":"7e118cb92fd7a9bae1785d9b88837cf7777cbd8cad2857a3e574e675b6ce1b95"}
Jul 20 19:27:01.435469 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:01.435432    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7lfs6" event={"ID":"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11","Type":"ContainerStarted","Data":"2b537828d39e3cb37c269311791dbb14f953609f1e0970eb3b520a9677aca7fb"}
Jul 20 19:27:04.218413 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.218376    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f7cd8cc88-56xvn"]
Jul 20 19:27:04.236585 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.236552    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.238970 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.238941    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7cd8cc88-56xvn"]
Jul 20 19:27:04.239598 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.239573    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 20 19:27:04.239737 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.239646    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 20 19:27:04.239737 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.239573    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 20 19:27:04.239737 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.239703    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 20 19:27:04.240346 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.240328    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 20 19:27:04.240413 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.240358    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jspd8\""
Jul 20 19:27:04.240587 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.240569    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 20 19:27:04.240695 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.240672    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 20 19:27:04.282148 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.282118    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdf8v\" (UniqueName: \"kubernetes.io/projected/6cc2dd31-ccc1-4497-85a7-657802445040-kube-api-access-bdf8v\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.282294 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.282150    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-service-ca\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.282294 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.282177    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-console-config\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.282294 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.282197    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-oauth-serving-cert\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.282490 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.282314    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-oauth-config\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.282490 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.282376    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-serving-cert\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.383647 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.383610    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-oauth-config\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.383647 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.383652    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-serving-cert\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.383839 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.383697    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdf8v\" (UniqueName: \"kubernetes.io/projected/6cc2dd31-ccc1-4497-85a7-657802445040-kube-api-access-bdf8v\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.383839 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.383717    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-service-ca\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.383839 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.383732    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-console-config\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.383839 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.383754    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-oauth-serving-cert\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.384493 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.384465    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-console-config\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.384614 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.384557    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-service-ca\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.384614 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.384579    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-oauth-serving-cert\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.386164 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.386137    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-oauth-config\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.386249 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.386213    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-serving-cert\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.398715 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.398688    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdf8v\" (UniqueName: \"kubernetes.io/projected/6cc2dd31-ccc1-4497-85a7-657802445040-kube-api-access-bdf8v\") pod \"console-5f7cd8cc88-56xvn\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") " pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.448433 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.448395    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7lfs6" event={"ID":"deddaa5c-cfd1-42bf-99e5-4a1f3b434a11","Type":"ContainerStarted","Data":"e63be4b952dfd108ff7ed45de69dc1ae8f2db83c3e42c05f2bd25a231b8ecacc"}
Jul 20 19:27:04.475314 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.475214    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7lfs6" podStartSLOduration=16.735679125 podStartE2EDuration="28.475199054s" totalImagesPullingTime="11.739519929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 19:26:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:27:04.474891382 +0000 UTC m=+276.560842382" watchObservedRunningTime="2026-07-20 19:27:04.475199054 +0000 UTC m=+276.561150077"
Jul 20 19:27:04.547436 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.547403    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:04.677607 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:04.677570    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7cd8cc88-56xvn"]
Jul 20 19:27:04.680145 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:27:04.680120    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc2dd31_ccc1_4497_85a7_657802445040.slice/crio-b9600fd336ee174d2f7cc248a9fc90eb1e56f0b84a85550d5bbcffc975049683 WatchSource:0}: Error finding container b9600fd336ee174d2f7cc248a9fc90eb1e56f0b84a85550d5bbcffc975049683: Status 404 returned error can't find the container with id b9600fd336ee174d2f7cc248a9fc90eb1e56f0b84a85550d5bbcffc975049683
Jul 20 19:27:05.450829 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:05.450782    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7cd8cc88-56xvn" event={"ID":"6cc2dd31-ccc1-4497-85a7-657802445040","Type":"ContainerStarted","Data":"b9600fd336ee174d2f7cc248a9fc90eb1e56f0b84a85550d5bbcffc975049683"}
Jul 20 19:27:07.450244 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:07.450211    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb"]
Jul 20 19:27:07.455139 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:07.455105    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb"
Jul 20 19:27:07.458211 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:07.458187    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-w58kt\""
Jul 20 19:27:07.458409 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:07.458385    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 20 19:27:07.473535 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:07.473510    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb"]
Jul 20 19:27:07.504533 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:07.504504    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/26599cb7-6a15-4ded-a7ee-14b884d2d9ee-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-zjjzb\" (UID: \"26599cb7-6a15-4ded-a7ee-14b884d2d9ee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb"
Jul 20 19:27:07.605728 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:07.605699    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/26599cb7-6a15-4ded-a7ee-14b884d2d9ee-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-zjjzb\" (UID: \"26599cb7-6a15-4ded-a7ee-14b884d2d9ee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb"
Jul 20 19:27:07.605917 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:27:07.605850    2550 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 20 19:27:07.605986 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:27:07.605935    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26599cb7-6a15-4ded-a7ee-14b884d2d9ee-tls-certificates podName:26599cb7-6a15-4ded-a7ee-14b884d2d9ee nodeName:}" failed. No retries permitted until 2026-07-20 19:27:08.105912245 +0000 UTC m=+280.191863237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/26599cb7-6a15-4ded-a7ee-14b884d2d9ee-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-zjjzb" (UID: "26599cb7-6a15-4ded-a7ee-14b884d2d9ee") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 20 19:27:08.110251 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:08.110166    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/26599cb7-6a15-4ded-a7ee-14b884d2d9ee-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-zjjzb\" (UID: \"26599cb7-6a15-4ded-a7ee-14b884d2d9ee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb"
Jul 20 19:27:08.112715 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:08.112692    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/26599cb7-6a15-4ded-a7ee-14b884d2d9ee-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-zjjzb\" (UID: \"26599cb7-6a15-4ded-a7ee-14b884d2d9ee\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb"
Jul 20 19:27:08.367513 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:08.367418    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb"
Jul 20 19:27:08.463149 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:08.463118    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7cd8cc88-56xvn" event={"ID":"6cc2dd31-ccc1-4497-85a7-657802445040","Type":"ContainerStarted","Data":"f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355"}
Jul 20 19:27:08.500621 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:08.500573    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f7cd8cc88-56xvn" podStartSLOduration=1.355314509 podStartE2EDuration="4.500557503s" totalImagesPullingTime="3.145242994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:27:08.493491665 +0000 UTC m=+280.579442663" watchObservedRunningTime="2026-07-20 19:27:08.500557503 +0000 UTC m=+280.586508500"
Jul 20 19:27:08.501408 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:08.501386    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb"]
Jul 20 19:27:08.503954 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:27:08.503928    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26599cb7_6a15_4ded_a7ee_14b884d2d9ee.slice/crio-d141ec602af9538031d2e77a151f62020eb611360098717bfb6ba08b55c9a516 WatchSource:0}: Error finding container d141ec602af9538031d2e77a151f62020eb611360098717bfb6ba08b55c9a516: Status 404 returned error can't find the container with id d141ec602af9538031d2e77a151f62020eb611360098717bfb6ba08b55c9a516
Jul 20 19:27:09.467645 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:09.467606    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb" event={"ID":"26599cb7-6a15-4ded-a7ee-14b884d2d9ee","Type":"ContainerStarted","Data":"d141ec602af9538031d2e77a151f62020eb611360098717bfb6ba08b55c9a516"}
Jul 20 19:27:10.104785 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:27:10.104745    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vlxp5" podUID="2fe76da7-11be-40ce-81e9-8e57306a184d"
Jul 20 19:27:10.472276 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:10.472227    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:27:10.472276 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:10.472250    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb" event={"ID":"26599cb7-6a15-4ded-a7ee-14b884d2d9ee","Type":"ContainerStarted","Data":"ca43b0d539d78af80f58ddb80570e2d8f5a362c10ee841100fcc2c9a6e67d546"}
Jul 20 19:27:10.472706 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:10.472432    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb"
Jul 20 19:27:10.476927 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:10.476906    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb"
Jul 20 19:27:10.487692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:10.487643    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-zjjzb" podStartSLOduration=2.41811191 podStartE2EDuration="3.487629525s" totalImagesPullingTime="1.069517615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:27:10.487050534 +0000 UTC m=+282.573001531" watchObservedRunningTime="2026-07-20 19:27:10.487629525 +0000 UTC m=+282.573580524"
Jul 20 19:27:13.249239 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:13.249192    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:27:13.251427 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:13.251406    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2fe76da7-11be-40ce-81e9-8e57306a184d-metrics-tls\") pod \"dns-default-vlxp5\" (UID: \"2fe76da7-11be-40ce-81e9-8e57306a184d\") " pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:27:13.475597 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:13.475560    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bsgbb\""
Jul 20 19:27:13.484226 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:13.484202    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:27:13.608409 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:13.608377    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vlxp5"]
Jul 20 19:27:13.611378 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:27:13.611351    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe76da7_11be_40ce_81e9_8e57306a184d.slice/crio-47e0cfc3f8a18f6157b35987e66a8db0ad70a8e1427bcb6ca0260354bf8f31da WatchSource:0}: Error finding container 47e0cfc3f8a18f6157b35987e66a8db0ad70a8e1427bcb6ca0260354bf8f31da: Status 404 returned error can't find the container with id 47e0cfc3f8a18f6157b35987e66a8db0ad70a8e1427bcb6ca0260354bf8f31da
Jul 20 19:27:14.486524 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:14.486484    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlxp5" event={"ID":"2fe76da7-11be-40ce-81e9-8e57306a184d","Type":"ContainerStarted","Data":"47e0cfc3f8a18f6157b35987e66a8db0ad70a8e1427bcb6ca0260354bf8f31da"}
Jul 20 19:27:14.547591 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:14.547561    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:14.547750 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:14.547619    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:14.552371 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:14.552350    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:15.491877 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:15.491787    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlxp5" event={"ID":"2fe76da7-11be-40ce-81e9-8e57306a184d","Type":"ContainerStarted","Data":"832aa6b46b0c82269c5b7d66a581164befc2e423a39e5fd4cc4a14d45ac7a959"}
Jul 20 19:27:15.491877 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:15.491838    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlxp5" event={"ID":"2fe76da7-11be-40ce-81e9-8e57306a184d","Type":"ContainerStarted","Data":"ee8384cb74cd5c05bc1755af583c0471f4b113d24cafa56bd418887e947ade26"}
Jul 20 19:27:15.496249 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:15.496227    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:15.509350 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:15.509299    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vlxp5" podStartSLOduration=250.982533064 podStartE2EDuration="4m12.509280339s" totalImagesPullingTime="1.526747275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:27:15.508342362 +0000 UTC m=+287.594293360" watchObservedRunningTime="2026-07-20 19:27:15.509280339 +0000 UTC m=+287.595231333"
Jul 20 19:27:16.495500 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:16.495472    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:27:18.041824 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.041789    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4hlp8"]
Jul 20 19:27:18.044816 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.044794    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.047825 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.047798    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 20 19:27:18.048067 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.047956    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 20 19:27:18.048823 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.048805    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sg4tx\""
Jul 20 19:27:18.048920 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.048858    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 20 19:27:18.048920 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.048868    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 20 19:27:18.050445 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.050428    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 20 19:27:18.050552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.050535    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 20 19:27:18.086770 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.086744    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-accelerators-collector-config\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.086885 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.086779    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-textfile\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.086885 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.086798    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-wtmp\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.086885 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.086815    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-tls\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.086885 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.086876    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.087010 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.086898    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6094ea31-c721-46b1-9209-e919faff4a00-root\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.087010 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.086912    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qw9k\" (UniqueName: \"kubernetes.io/projected/6094ea31-c721-46b1-9209-e919faff4a00-kube-api-access-7qw9k\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.087010 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.086934    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6094ea31-c721-46b1-9209-e919faff4a00-sys\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.087010 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.086978    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6094ea31-c721-46b1-9209-e919faff4a00-metrics-client-ca\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.187846 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.187813    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6094ea31-c721-46b1-9209-e919faff4a00-root\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.187846 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.187847    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qw9k\" (UniqueName: \"kubernetes.io/projected/6094ea31-c721-46b1-9209-e919faff4a00-kube-api-access-7qw9k\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188050 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.187871    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6094ea31-c721-46b1-9209-e919faff4a00-sys\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188050 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.187918    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6094ea31-c721-46b1-9209-e919faff4a00-metrics-client-ca\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188050 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.187931    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6094ea31-c721-46b1-9209-e919faff4a00-root\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188050 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.187936    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-accelerators-collector-config\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188276 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.188055    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-textfile\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188276 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.187975    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6094ea31-c721-46b1-9209-e919faff4a00-sys\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188276 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.188083    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-wtmp\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188276 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.188115    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-tls\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188276 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.188147    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188276 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.188227    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-wtmp\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188551 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.188376    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-textfile\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188673 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.188651    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-accelerators-collector-config\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.188723 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.188657    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6094ea31-c721-46b1-9209-e919faff4a00-metrics-client-ca\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.190324 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.190303    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-tls\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.190413 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.190398    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6094ea31-c721-46b1-9209-e919faff4a00-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.199378 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.199357    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qw9k\" (UniqueName: \"kubernetes.io/projected/6094ea31-c721-46b1-9209-e919faff4a00-kube-api-access-7qw9k\") pod \"node-exporter-4hlp8\" (UID: \"6094ea31-c721-46b1-9209-e919faff4a00\") " pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.353749 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.353653    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4hlp8"
Jul 20 19:27:18.364012 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:27:18.363989    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6094ea31_c721_46b1_9209_e919faff4a00.slice/crio-93180ff98908b3d305512ea559a693688c720768597723ca7fe36c507619044b WatchSource:0}: Error finding container 93180ff98908b3d305512ea559a693688c720768597723ca7fe36c507619044b: Status 404 returned error can't find the container with id 93180ff98908b3d305512ea559a693688c720768597723ca7fe36c507619044b
Jul 20 19:27:18.502149 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:18.502118    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4hlp8" event={"ID":"6094ea31-c721-46b1-9209-e919faff4a00","Type":"ContainerStarted","Data":"93180ff98908b3d305512ea559a693688c720768597723ca7fe36c507619044b"}
Jul 20 19:27:20.446222 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:27:20.446179    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6bdk5" podUID="cd899fed-afcf-484c-89bf-28566c07b2d2"
Jul 20 19:27:20.509577 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:20.509547    2550 generic.go:356] "Generic (PLEG): container finished" podID="6094ea31-c721-46b1-9209-e919faff4a00" containerID="f362b11234980fa79e73f4d91f068b476d993594f00d6a3ef400662858a2cde2" exitCode=0
Jul 20 19:27:20.509762 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:20.509637    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4hlp8" event={"ID":"6094ea31-c721-46b1-9209-e919faff4a00","Type":"ContainerDied","Data":"f362b11234980fa79e73f4d91f068b476d993594f00d6a3ef400662858a2cde2"}
Jul 20 19:27:21.515308 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:21.515246    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4hlp8" event={"ID":"6094ea31-c721-46b1-9209-e919faff4a00","Type":"ContainerStarted","Data":"99f0726abef421142e468c2f90d261b2f43d688c406ddbd2d7ab73fd688d7c26"}
Jul 20 19:27:21.515688 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:21.515332    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4hlp8" event={"ID":"6094ea31-c721-46b1-9209-e919faff4a00","Type":"ContainerStarted","Data":"7267dcdbe6b86416feee3554136b53dfedce5c15e3de9c7e5ab8d2d2c6584940"}
Jul 20 19:27:21.536187 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:21.536140    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4hlp8" podStartSLOduration=2.382655488 podStartE2EDuration="3.53612665s" totalImagesPullingTime="1.088278889s" totalInitContainerRuntime="65.192273ms" isStatefulPod=true podCreationTimestamp="2026-07-20 19:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:27:21.534964309 +0000 UTC m=+293.620915301" watchObservedRunningTime="2026-07-20 19:27:21.53612665 +0000 UTC m=+293.622077648"
Jul 20 19:27:21.910805 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:21.910774    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7cd8cc88-56xvn"]
Jul 20 19:27:25.980771 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:25.980736    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ff5d887fd-k6bt9"]
Jul 20 19:27:25.984013 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:25.983994    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:25.991561 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:25.991537    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 20 19:27:25.996185 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:25.996086    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff5d887fd-k6bt9"]
Jul 20 19:27:26.050052 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.050019    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-service-ca\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.050198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.050085    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-trusted-ca-bundle\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.050198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.050152    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-oauth-serving-cert\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.050198 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.050183    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-oauth-config\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.050379 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.050213    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6grq\" (UniqueName: \"kubernetes.io/projected/d394b68d-2255-4a02-8a1f-29d3e5002dbd-kube-api-access-v6grq\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.050379 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.050234    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-config\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.050379 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.050297    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-serving-cert\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.151443 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.151413    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-service-ca\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.151596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.151465    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-trusted-ca-bundle\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.151596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.151585    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-oauth-serving-cert\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.151711 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.151630    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-oauth-config\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.151711 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.151676    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6grq\" (UniqueName: \"kubernetes.io/projected/d394b68d-2255-4a02-8a1f-29d3e5002dbd-kube-api-access-v6grq\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.151711 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.151704    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-config\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.151855 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.151738    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-serving-cert\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.152301 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.152248    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-service-ca\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.152420 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.152310    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-oauth-serving-cert\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.152484 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.152416    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-trusted-ca-bundle\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.152484 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.152447    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-config\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.154078 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.154051    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-oauth-config\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.154200 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.154178    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-serving-cert\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.160536 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.160519    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6grq\" (UniqueName: \"kubernetes.io/projected/d394b68d-2255-4a02-8a1f-29d3e5002dbd-kube-api-access-v6grq\") pod \"console-6ff5d887fd-k6bt9\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") " pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.292537 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.292460    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:26.417502 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.417340    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff5d887fd-k6bt9"]
Jul 20 19:27:26.419890 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:27:26.419866    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd394b68d_2255_4a02_8a1f_29d3e5002dbd.slice/crio-f0c083e0b7705bcd135c5c81f9ab1f80f8f804b93df7af9077f53533624e0023 WatchSource:0}: Error finding container f0c083e0b7705bcd135c5c81f9ab1f80f8f804b93df7af9077f53533624e0023: Status 404 returned error can't find the container with id f0c083e0b7705bcd135c5c81f9ab1f80f8f804b93df7af9077f53533624e0023
Jul 20 19:27:26.500463 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.500438    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vlxp5"
Jul 20 19:27:26.547326 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.546337    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff5d887fd-k6bt9" event={"ID":"d394b68d-2255-4a02-8a1f-29d3e5002dbd","Type":"ContainerStarted","Data":"65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6"}
Jul 20 19:27:26.547326 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.546399    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff5d887fd-k6bt9" event={"ID":"d394b68d-2255-4a02-8a1f-29d3e5002dbd","Type":"ContainerStarted","Data":"f0c083e0b7705bcd135c5c81f9ab1f80f8f804b93df7af9077f53533624e0023"}
Jul 20 19:27:26.575423 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:26.571573    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6ff5d887fd-k6bt9" podStartSLOduration=1.5715524109999999 podStartE2EDuration="1.571552411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:27:26.571135526 +0000 UTC m=+298.657086527" watchObservedRunningTime="2026-07-20 19:27:26.571552411 +0000 UTC m=+298.657503410"
Jul 20 19:27:28.329388 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:28.329352    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:27:28.329873 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:28.329662    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:27:28.335579 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:28.335559    2550 kubelet.go:1648] "Image garbage collection succeeded"
Jul 20 19:27:35.444447 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:35.444398    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:27:36.293578 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:36.293541    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:36.293771 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:36.293613    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:36.298472 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:36.298448    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:36.579998 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:36.579916    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:27:46.930774 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:46.930714    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f7cd8cc88-56xvn" podUID="6cc2dd31-ccc1-4497-85a7-657802445040" containerName="console" containerID="cri-o://f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355" gracePeriod=15
Jul 20 19:27:47.163916 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.163892    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:47.328000 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.327912    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-serving-cert\") pod \"6cc2dd31-ccc1-4497-85a7-657802445040\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") "
Jul 20 19:27:47.328000 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.327953    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-service-ca\") pod \"6cc2dd31-ccc1-4497-85a7-657802445040\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") "
Jul 20 19:27:47.328000 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.327981    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-console-config\") pod \"6cc2dd31-ccc1-4497-85a7-657802445040\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") "
Jul 20 19:27:47.328207 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.328176    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-oauth-config\") pod \"6cc2dd31-ccc1-4497-85a7-657802445040\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") "
Jul 20 19:27:47.328250 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.328212    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdf8v\" (UniqueName: \"kubernetes.io/projected/6cc2dd31-ccc1-4497-85a7-657802445040-kube-api-access-bdf8v\") pod \"6cc2dd31-ccc1-4497-85a7-657802445040\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") "
Jul 20 19:27:47.328352 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.328317    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-oauth-serving-cert\") pod \"6cc2dd31-ccc1-4497-85a7-657802445040\" (UID: \"6cc2dd31-ccc1-4497-85a7-657802445040\") "
Jul 20 19:27:47.328406 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.328351    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-service-ca" (OuterVolumeSpecName: "service-ca") pod "6cc2dd31-ccc1-4497-85a7-657802445040" (UID: "6cc2dd31-ccc1-4497-85a7-657802445040"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:27:47.328479 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.328452    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-console-config" (OuterVolumeSpecName: "console-config") pod "6cc2dd31-ccc1-4497-85a7-657802445040" (UID: "6cc2dd31-ccc1-4497-85a7-657802445040"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:27:47.328579 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.328561    2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:27:47.328631 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.328584    2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-console-config\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:27:47.328720 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.328699    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6cc2dd31-ccc1-4497-85a7-657802445040" (UID: "6cc2dd31-ccc1-4497-85a7-657802445040"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:27:47.330427 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.330403    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6cc2dd31-ccc1-4497-85a7-657802445040" (UID: "6cc2dd31-ccc1-4497-85a7-657802445040"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:27:47.330529 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.330451    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc2dd31-ccc1-4497-85a7-657802445040-kube-api-access-bdf8v" (OuterVolumeSpecName: "kube-api-access-bdf8v") pod "6cc2dd31-ccc1-4497-85a7-657802445040" (UID: "6cc2dd31-ccc1-4497-85a7-657802445040"). InnerVolumeSpecName "kube-api-access-bdf8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:27:47.330529 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.330478    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6cc2dd31-ccc1-4497-85a7-657802445040" (UID: "6cc2dd31-ccc1-4497-85a7-657802445040"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:27:47.428898 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.428867    2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-oauth-config\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:27:47.428898 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.428894    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdf8v\" (UniqueName: \"kubernetes.io/projected/6cc2dd31-ccc1-4497-85a7-657802445040-kube-api-access-bdf8v\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:27:47.428898 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.428904    2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cc2dd31-ccc1-4497-85a7-657802445040-oauth-serving-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:27:47.429104 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.428915    2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cc2dd31-ccc1-4497-85a7-657802445040-console-serving-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:27:47.611388 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.611301    2550 generic.go:356] "Generic (PLEG): container finished" podID="6cc2dd31-ccc1-4497-85a7-657802445040" containerID="f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355" exitCode=0
Jul 20 19:27:47.611388 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.611350    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7cd8cc88-56xvn"
Jul 20 19:27:47.611388 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.611365    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7cd8cc88-56xvn" event={"ID":"6cc2dd31-ccc1-4497-85a7-657802445040","Type":"ContainerDied","Data":"f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355"}
Jul 20 19:27:47.611601 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.611405    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7cd8cc88-56xvn" event={"ID":"6cc2dd31-ccc1-4497-85a7-657802445040","Type":"ContainerDied","Data":"b9600fd336ee174d2f7cc248a9fc90eb1e56f0b84a85550d5bbcffc975049683"}
Jul 20 19:27:47.611601 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.611424    2550 scope.go:117] "RemoveContainer" containerID="f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355"
Jul 20 19:27:47.619984 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.619965    2550 scope.go:117] "RemoveContainer" containerID="f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355"
Jul 20 19:27:47.620236 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:27:47.620220    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355\": container with ID starting with f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355 not found: ID does not exist" containerID="f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355"
Jul 20 19:27:47.620326 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.620243    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355"} err="failed to get container status \"f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355\": rpc error: code = NotFound desc = could not find container \"f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355\": container with ID starting with f5e2dccc26eadf03e8203a31a1fe4fa23f7fec499186f8c2689665d58972f355 not found: ID does not exist"
Jul 20 19:27:47.632371 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.632346    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7cd8cc88-56xvn"]
Jul 20 19:27:47.636304 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:47.636279    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f7cd8cc88-56xvn"]
Jul 20 19:27:48.448314 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:48.448283    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cc2dd31-ccc1-4497-85a7-657802445040" path="/var/lib/kubelet/pods/6cc2dd31-ccc1-4497-85a7-657802445040/volumes"
Jul 20 19:27:52.472382 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:27:52.472343    2550 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" podUID="bf1b980a-5805-43aa-8e24-82433b1e186b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 20 19:28:02.472069 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:02.472033    2550 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" podUID="bf1b980a-5805-43aa-8e24-82433b1e186b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 20 19:28:12.472639 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:12.472591    2550 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" podUID="bf1b980a-5805-43aa-8e24-82433b1e186b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 20 19:28:12.473128 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:12.472680    2550 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd"
Jul 20 19:28:12.473287 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:12.473246    2550 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"640ead496d2d752660be7a8d83d6a990166a18c55fa1da8038b446d6cae16c49"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 20 19:28:12.473366 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:12.473314    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" podUID="bf1b980a-5805-43aa-8e24-82433b1e186b" containerName="service-proxy" containerID="cri-o://640ead496d2d752660be7a8d83d6a990166a18c55fa1da8038b446d6cae16c49" gracePeriod=30
Jul 20 19:28:12.694167 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:12.694126    2550 generic.go:356] "Generic (PLEG): container finished" podID="bf1b980a-5805-43aa-8e24-82433b1e186b" containerID="640ead496d2d752660be7a8d83d6a990166a18c55fa1da8038b446d6cae16c49" exitCode=2
Jul 20 19:28:12.694360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:12.694187    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" event={"ID":"bf1b980a-5805-43aa-8e24-82433b1e186b","Type":"ContainerDied","Data":"640ead496d2d752660be7a8d83d6a990166a18c55fa1da8038b446d6cae16c49"}
Jul 20 19:28:12.694360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:12.694240    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b875bffcb-tt7bd" event={"ID":"bf1b980a-5805-43aa-8e24-82433b1e186b","Type":"ContainerStarted","Data":"f9e861efbc7a5d0fd2119c1055fdd1071504a7e507d511b226cdeade31b1e622"}
Jul 20 19:28:41.316402 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:41.316364    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:28:41.318589 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:41.318570    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd899fed-afcf-484c-89bf-28566c07b2d2-metrics-certs\") pod \"network-metrics-daemon-6bdk5\" (UID: \"cd899fed-afcf-484c-89bf-28566c07b2d2\") " pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:28:41.447324 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:41.447291    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gzsbm\""
Jul 20 19:28:41.455119 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:41.455097    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bdk5"
Jul 20 19:28:41.575563 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:41.575539    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6bdk5"]
Jul 20 19:28:41.578051 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:28:41.578026    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd899fed_afcf_484c_89bf_28566c07b2d2.slice/crio-5fc10be93062f96e110b9bc0b43e7a9aed0d50cfff5e031b37397b86d8cf5f01 WatchSource:0}: Error finding container 5fc10be93062f96e110b9bc0b43e7a9aed0d50cfff5e031b37397b86d8cf5f01: Status 404 returned error can't find the container with id 5fc10be93062f96e110b9bc0b43e7a9aed0d50cfff5e031b37397b86d8cf5f01
Jul 20 19:28:41.579940 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:41.579924    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 19:28:41.788570 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:41.788537    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6bdk5" event={"ID":"cd899fed-afcf-484c-89bf-28566c07b2d2","Type":"ContainerStarted","Data":"5fc10be93062f96e110b9bc0b43e7a9aed0d50cfff5e031b37397b86d8cf5f01"}
Jul 20 19:28:42.793234 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:42.793145    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6bdk5" event={"ID":"cd899fed-afcf-484c-89bf-28566c07b2d2","Type":"ContainerStarted","Data":"adbf683350a782d4bc8d0c40192c2f4f7f233114b2663b0902d158f30937a9d5"}
Jul 20 19:28:42.793234 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:42.793203    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6bdk5" event={"ID":"cd899fed-afcf-484c-89bf-28566c07b2d2","Type":"ContainerStarted","Data":"463e81a574643351a95b93e3aa2e877b7d57247667756580d50347d72271d8dc"}
Jul 20 19:28:42.812964 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:42.812916    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6bdk5" podStartSLOduration=373.956542431 podStartE2EDuration="6m14.812901169s" totalImagesPullingTime="856.358738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:28:42.81205839 +0000 UTC m=+374.898009390" watchObservedRunningTime="2026-07-20 19:28:42.812901169 +0000 UTC m=+374.898852168"
Jul 20 19:28:46.109360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.109326    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ddbcc576b-8jmmz"]
Jul 20 19:28:46.109744 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.109560    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cc2dd31-ccc1-4497-85a7-657802445040" containerName="console"
Jul 20 19:28:46.109744 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.109571    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cc2dd31-ccc1-4497-85a7-657802445040" containerName="console"
Jul 20 19:28:46.109744 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.109632    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cc2dd31-ccc1-4497-85a7-657802445040" containerName="console"
Jul 20 19:28:46.112356 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.112339    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.123729 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.123707    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ddbcc576b-8jmmz"]
Jul 20 19:28:46.151535 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.151500    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-oauth-config\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.151664 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.151549    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-trusted-ca-bundle\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.151664 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.151575    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-oauth-serving-cert\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.151664 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.151644    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-service-ca\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.151772 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.151699    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7s7\" (UniqueName: \"kubernetes.io/projected/186330e1-15f4-4ff6-9c69-2e7166a21249-kube-api-access-7z7s7\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.151772 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.151722    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-console-config\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.151772 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.151742    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-serving-cert\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.252670 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.252636    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-console-config\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.252670 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.252674    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-serving-cert\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.252873 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.252699    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-oauth-config\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.252873 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.252720    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-trusted-ca-bundle\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.252873 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.252735    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-oauth-serving-cert\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.252873 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.252758    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-service-ca\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.252873 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.252787    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7s7\" (UniqueName: \"kubernetes.io/projected/186330e1-15f4-4ff6-9c69-2e7166a21249-kube-api-access-7z7s7\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.253496 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.253469    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-service-ca\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.253611 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.253504    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-console-config\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.253611 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.253545    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-oauth-serving-cert\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.253745 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.253719    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-trusted-ca-bundle\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.255192 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.255170    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-oauth-config\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.255296 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.255281    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-serving-cert\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.261321 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.261301    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7s7\" (UniqueName: \"kubernetes.io/projected/186330e1-15f4-4ff6-9c69-2e7166a21249-kube-api-access-7z7s7\") pod \"console-5ddbcc576b-8jmmz\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") " pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.421070 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.421038    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:28:46.551077 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.551043    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ddbcc576b-8jmmz"]
Jul 20 19:28:46.553647 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:28:46.553625    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186330e1_15f4_4ff6_9c69_2e7166a21249.slice/crio-de5bca2f07e99c45626328fa25d3a15f46d14b5e8c24a1375fbeb00ac334eeb6 WatchSource:0}: Error finding container de5bca2f07e99c45626328fa25d3a15f46d14b5e8c24a1375fbeb00ac334eeb6: Status 404 returned error can't find the container with id de5bca2f07e99c45626328fa25d3a15f46d14b5e8c24a1375fbeb00ac334eeb6
Jul 20 19:28:46.808488 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.808406    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ddbcc576b-8jmmz" event={"ID":"186330e1-15f4-4ff6-9c69-2e7166a21249","Type":"ContainerStarted","Data":"baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec"}
Jul 20 19:28:46.808488 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.808451    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ddbcc576b-8jmmz" event={"ID":"186330e1-15f4-4ff6-9c69-2e7166a21249","Type":"ContainerStarted","Data":"de5bca2f07e99c45626328fa25d3a15f46d14b5e8c24a1375fbeb00ac334eeb6"}
Jul 20 19:28:46.827365 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:46.827316    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ddbcc576b-8jmmz" podStartSLOduration=0.827299553 podStartE2EDuration="827.299553ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:28:46.826068735 +0000 UTC m=+378.912019979" watchObservedRunningTime="2026-07-20 19:28:46.827299553 +0000 UTC m=+378.913250551"
Jul 20 19:28:54.852823 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:54.852775    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ddbcc576b-8jmmz"]
Jul 20 19:28:54.885507 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:54.885479    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55f578c898-sh7xw"]
Jul 20 19:28:54.888460 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:54.888444    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:54.899758 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:54.899734    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f578c898-sh7xw"]
Jul 20 19:28:55.016486 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.016449    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g988s\" (UniqueName: \"kubernetes.io/projected/8d1cf28d-721a-435c-bab3-ff1301f9eec7-kube-api-access-g988s\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.016676 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.016502    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-oauth-serving-cert\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.016676 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.016562    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-config\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.016676 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.016605    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-service-ca\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.016676 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.016629    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-oauth-config\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.016676 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.016650    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-trusted-ca-bundle\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.016847 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.016681    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-serving-cert\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.117869 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.117790    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g988s\" (UniqueName: \"kubernetes.io/projected/8d1cf28d-721a-435c-bab3-ff1301f9eec7-kube-api-access-g988s\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.117869 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.117848    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-oauth-serving-cert\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.117869 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.117869    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-config\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.118047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.117895    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-service-ca\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.118047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.117915    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-oauth-config\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.118047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.117933    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-trusted-ca-bundle\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.118047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.117951    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-serving-cert\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.118642 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.118608    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-oauth-serving-cert\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.118782 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.118757    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-service-ca\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.118835 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.118797    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-config\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.118889 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.118874    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-trusted-ca-bundle\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.120472 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.120447    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-serving-cert\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.120562 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.120476    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-oauth-config\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.125868 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.125843    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g988s\" (UniqueName: \"kubernetes.io/projected/8d1cf28d-721a-435c-bab3-ff1301f9eec7-kube-api-access-g988s\") pod \"console-55f578c898-sh7xw\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") " pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.197607 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.197586    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:28:55.321325 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.321224    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f578c898-sh7xw"]
Jul 20 19:28:55.323812 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:28:55.323788    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1cf28d_721a_435c_bab3_ff1301f9eec7.slice/crio-bbf9f2beada72d8df4b62ef518010921ac61ad796e1231cb517044172bc2c188 WatchSource:0}: Error finding container bbf9f2beada72d8df4b62ef518010921ac61ad796e1231cb517044172bc2c188: Status 404 returned error can't find the container with id bbf9f2beada72d8df4b62ef518010921ac61ad796e1231cb517044172bc2c188
Jul 20 19:28:55.847570 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.847534    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f578c898-sh7xw" event={"ID":"8d1cf28d-721a-435c-bab3-ff1301f9eec7","Type":"ContainerStarted","Data":"8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861"}
Jul 20 19:28:55.847922 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.847594    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f578c898-sh7xw" event={"ID":"8d1cf28d-721a-435c-bab3-ff1301f9eec7","Type":"ContainerStarted","Data":"bbf9f2beada72d8df4b62ef518010921ac61ad796e1231cb517044172bc2c188"}
Jul 20 19:28:55.867344 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:55.867294    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55f578c898-sh7xw" podStartSLOduration=1.867277249 podStartE2EDuration="1.867277249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:28:55.866521301 +0000 UTC m=+387.952472325" watchObservedRunningTime="2026-07-20 19:28:55.867277249 +0000 UTC m=+387.953228243"
Jul 20 19:28:56.422096 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:28:56.422063    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:29:05.197892 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:05.197844    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:29:05.198438 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:05.197916    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:29:05.202573 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:05.202553    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:29:05.886277 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:05.886235    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:29:05.942025 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:05.941995    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff5d887fd-k6bt9"]
Jul 20 19:29:19.871874 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:19.871829    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5ddbcc576b-8jmmz" podUID="186330e1-15f4-4ff6-9c69-2e7166a21249" containerName="console" containerID="cri-o://baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec" gracePeriod=15
Jul 20 19:29:20.102543 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.102521    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:29:20.198586 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.198501    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-service-ca\") pod \"186330e1-15f4-4ff6-9c69-2e7166a21249\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") "
Jul 20 19:29:20.198586 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.198543    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z7s7\" (UniqueName: \"kubernetes.io/projected/186330e1-15f4-4ff6-9c69-2e7166a21249-kube-api-access-7z7s7\") pod \"186330e1-15f4-4ff6-9c69-2e7166a21249\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") "
Jul 20 19:29:20.198586 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.198569    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-oauth-config\") pod \"186330e1-15f4-4ff6-9c69-2e7166a21249\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") "
Jul 20 19:29:20.198919 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.198636    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-oauth-serving-cert\") pod \"186330e1-15f4-4ff6-9c69-2e7166a21249\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") "
Jul 20 19:29:20.198919 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.198651    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-trusted-ca-bundle\") pod \"186330e1-15f4-4ff6-9c69-2e7166a21249\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") "
Jul 20 19:29:20.198919 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.198668    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-serving-cert\") pod \"186330e1-15f4-4ff6-9c69-2e7166a21249\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") "
Jul 20 19:29:20.198919 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.198686    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-console-config\") pod \"186330e1-15f4-4ff6-9c69-2e7166a21249\" (UID: \"186330e1-15f4-4ff6-9c69-2e7166a21249\") "
Jul 20 19:29:20.199136 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.199042    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "186330e1-15f4-4ff6-9c69-2e7166a21249" (UID: "186330e1-15f4-4ff6-9c69-2e7166a21249"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:29:20.199136 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.199118    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "186330e1-15f4-4ff6-9c69-2e7166a21249" (UID: "186330e1-15f4-4ff6-9c69-2e7166a21249"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:29:20.199238 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.199126    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-console-config" (OuterVolumeSpecName: "console-config") pod "186330e1-15f4-4ff6-9c69-2e7166a21249" (UID: "186330e1-15f4-4ff6-9c69-2e7166a21249"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:29:20.199238 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.199195    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-service-ca" (OuterVolumeSpecName: "service-ca") pod "186330e1-15f4-4ff6-9c69-2e7166a21249" (UID: "186330e1-15f4-4ff6-9c69-2e7166a21249"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:29:20.200706 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.200686    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "186330e1-15f4-4ff6-9c69-2e7166a21249" (UID: "186330e1-15f4-4ff6-9c69-2e7166a21249"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:29:20.200804 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.200724    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186330e1-15f4-4ff6-9c69-2e7166a21249-kube-api-access-7z7s7" (OuterVolumeSpecName: "kube-api-access-7z7s7") pod "186330e1-15f4-4ff6-9c69-2e7166a21249" (UID: "186330e1-15f4-4ff6-9c69-2e7166a21249"). InnerVolumeSpecName "kube-api-access-7z7s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:29:20.200804 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.200791    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "186330e1-15f4-4ff6-9c69-2e7166a21249" (UID: "186330e1-15f4-4ff6-9c69-2e7166a21249"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:29:20.300134 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.300097    2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-oauth-serving-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:20.300134 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.300129    2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-trusted-ca-bundle\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:20.300134 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.300139    2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-serving-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:20.300374 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.300148    2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-console-config\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:20.300374 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.300157    2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/186330e1-15f4-4ff6-9c69-2e7166a21249-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:20.300374 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.300165    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7s7\" (UniqueName: \"kubernetes.io/projected/186330e1-15f4-4ff6-9c69-2e7166a21249-kube-api-access-7z7s7\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:20.300374 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.300173    2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/186330e1-15f4-4ff6-9c69-2e7166a21249-console-oauth-config\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:20.937839 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.937807    2550 generic.go:356] "Generic (PLEG): container finished" podID="186330e1-15f4-4ff6-9c69-2e7166a21249" containerID="baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec" exitCode=0
Jul 20 19:29:20.938246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.937853    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ddbcc576b-8jmmz"
Jul 20 19:29:20.938246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.937855    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ddbcc576b-8jmmz" event={"ID":"186330e1-15f4-4ff6-9c69-2e7166a21249","Type":"ContainerDied","Data":"baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec"}
Jul 20 19:29:20.938246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.937973    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ddbcc576b-8jmmz" event={"ID":"186330e1-15f4-4ff6-9c69-2e7166a21249","Type":"ContainerDied","Data":"de5bca2f07e99c45626328fa25d3a15f46d14b5e8c24a1375fbeb00ac334eeb6"}
Jul 20 19:29:20.938246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.938003    2550 scope.go:117] "RemoveContainer" containerID="baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec"
Jul 20 19:29:20.945890 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.945871    2550 scope.go:117] "RemoveContainer" containerID="baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec"
Jul 20 19:29:20.946132 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:29:20.946113    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec\": container with ID starting with baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec not found: ID does not exist" containerID="baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec"
Jul 20 19:29:20.946175 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.946141    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec"} err="failed to get container status \"baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec\": rpc error: code = NotFound desc = could not find container \"baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec\": container with ID starting with baf6c5aaeea321b1346333cf81999bcb856b9974b12ba62371e9c4fd45d32dec not found: ID does not exist"
Jul 20 19:29:20.953784 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.953764    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ddbcc576b-8jmmz"]
Jul 20 19:29:20.956319 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:20.956288    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ddbcc576b-8jmmz"]
Jul 20 19:29:22.450031 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:22.449995    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="186330e1-15f4-4ff6-9c69-2e7166a21249" path="/var/lib/kubelet/pods/186330e1-15f4-4ff6-9c69-2e7166a21249/volumes"
Jul 20 19:29:30.961729 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:30.961685    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6ff5d887fd-k6bt9" podUID="d394b68d-2255-4a02-8a1f-29d3e5002dbd" containerName="console" containerID="cri-o://65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6" gracePeriod=15
Jul 20 19:29:31.205383 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.205361    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:29:31.281007 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.280924    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-oauth-serving-cert\") pod \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") "
Jul 20 19:29:31.281007 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.280986    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6grq\" (UniqueName: \"kubernetes.io/projected/d394b68d-2255-4a02-8a1f-29d3e5002dbd-kube-api-access-v6grq\") pod \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") "
Jul 20 19:29:31.281188 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.281018    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-trusted-ca-bundle\") pod \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") "
Jul 20 19:29:31.281236 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.281207    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-config\") pod \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") "
Jul 20 19:29:31.281302 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.281285    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-serving-cert\") pod \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") "
Jul 20 19:29:31.281347 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.281329    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-service-ca\") pod \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") "
Jul 20 19:29:31.281388 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.281353    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-oauth-config\") pod \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\" (UID: \"d394b68d-2255-4a02-8a1f-29d3e5002dbd\") "
Jul 20 19:29:31.281388 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.281374    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d394b68d-2255-4a02-8a1f-29d3e5002dbd" (UID: "d394b68d-2255-4a02-8a1f-29d3e5002dbd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:29:31.281485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.281372    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d394b68d-2255-4a02-8a1f-29d3e5002dbd" (UID: "d394b68d-2255-4a02-8a1f-29d3e5002dbd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:29:31.281580 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.281562    2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-trusted-ca-bundle\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:31.281662 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.281587    2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-oauth-serving-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:31.281662 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.281618    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-config" (OuterVolumeSpecName: "console-config") pod "d394b68d-2255-4a02-8a1f-29d3e5002dbd" (UID: "d394b68d-2255-4a02-8a1f-29d3e5002dbd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:29:31.281774 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.281748    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-service-ca" (OuterVolumeSpecName: "service-ca") pod "d394b68d-2255-4a02-8a1f-29d3e5002dbd" (UID: "d394b68d-2255-4a02-8a1f-29d3e5002dbd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:29:31.283206 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.283181    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d394b68d-2255-4a02-8a1f-29d3e5002dbd-kube-api-access-v6grq" (OuterVolumeSpecName: "kube-api-access-v6grq") pod "d394b68d-2255-4a02-8a1f-29d3e5002dbd" (UID: "d394b68d-2255-4a02-8a1f-29d3e5002dbd"). InnerVolumeSpecName "kube-api-access-v6grq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:29:31.283400 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.283337    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d394b68d-2255-4a02-8a1f-29d3e5002dbd" (UID: "d394b68d-2255-4a02-8a1f-29d3e5002dbd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:29:31.283400 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.283371    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d394b68d-2255-4a02-8a1f-29d3e5002dbd" (UID: "d394b68d-2255-4a02-8a1f-29d3e5002dbd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:29:31.382663 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.382615    2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-serving-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:31.382663 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.382657    2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:31.382663 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.382666    2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-oauth-config\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:31.382663 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.382675    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6grq\" (UniqueName: \"kubernetes.io/projected/d394b68d-2255-4a02-8a1f-29d3e5002dbd-kube-api-access-v6grq\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:31.382918 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.382685    2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d394b68d-2255-4a02-8a1f-29d3e5002dbd-console-config\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:29:31.977167 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.977128    2550 generic.go:356] "Generic (PLEG): container finished" podID="d394b68d-2255-4a02-8a1f-29d3e5002dbd" containerID="65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6" exitCode=0
Jul 20 19:29:31.977591 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.977180    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff5d887fd-k6bt9"
Jul 20 19:29:31.977591 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.977190    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff5d887fd-k6bt9" event={"ID":"d394b68d-2255-4a02-8a1f-29d3e5002dbd","Type":"ContainerDied","Data":"65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6"}
Jul 20 19:29:31.977591 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.977239    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff5d887fd-k6bt9" event={"ID":"d394b68d-2255-4a02-8a1f-29d3e5002dbd","Type":"ContainerDied","Data":"f0c083e0b7705bcd135c5c81f9ab1f80f8f804b93df7af9077f53533624e0023"}
Jul 20 19:29:31.977591 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.977286    2550 scope.go:117] "RemoveContainer" containerID="65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6"
Jul 20 19:29:31.986250 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.986224    2550 scope.go:117] "RemoveContainer" containerID="65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6"
Jul 20 19:29:31.986536 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:29:31.986512    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6\": container with ID starting with 65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6 not found: ID does not exist" containerID="65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6"
Jul 20 19:29:31.986592 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:31.986544    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6"} err="failed to get container status \"65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6\": rpc error: code = NotFound desc = could not find container \"65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6\": container with ID starting with 65a6aeba3afe6c93a1cce30ea1cb2f94a333ec1a1bf4bdaf6be5cdcdacb2baa6 not found: ID does not exist"
Jul 20 19:29:32.000619 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:32.000588    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff5d887fd-k6bt9"]
Jul 20 19:29:32.004798 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:32.004777    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6ff5d887fd-k6bt9"]
Jul 20 19:29:32.449369 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:29:32.449342    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d394b68d-2255-4a02-8a1f-29d3e5002dbd" path="/var/lib/kubelet/pods/d394b68d-2255-4a02-8a1f-29d3e5002dbd/volumes"
Jul 20 19:30:11.288761 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.288680    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp"]
Jul 20 19:30:11.289216 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.288929    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d394b68d-2255-4a02-8a1f-29d3e5002dbd" containerName="console"
Jul 20 19:30:11.289216 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.288961    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d394b68d-2255-4a02-8a1f-29d3e5002dbd" containerName="console"
Jul 20 19:30:11.289216 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.288977    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="186330e1-15f4-4ff6-9c69-2e7166a21249" containerName="console"
Jul 20 19:30:11.289216 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.288982    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="186330e1-15f4-4ff6-9c69-2e7166a21249" containerName="console"
Jul 20 19:30:11.289216 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.289028    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d394b68d-2255-4a02-8a1f-29d3e5002dbd" containerName="console"
Jul 20 19:30:11.289216 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.289037    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="186330e1-15f4-4ff6-9c69-2e7166a21249" containerName="console"
Jul 20 19:30:11.291816 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.291800    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp"
Jul 20 19:30:11.294190 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.294166    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 20 19:30:11.294190 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.294167    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 20 19:30:11.294384 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.294325    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kp2c6\""
Jul 20 19:30:11.308065 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.308004    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp"]
Jul 20 19:30:11.374844 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.374805    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh2g9\" (UniqueName: \"kubernetes.io/projected/b995d863-4e5c-487b-84a8-c09faf07bc36-kube-api-access-wh2g9\") pod \"cert-manager-operator-controller-manager-55c444bf86-vb2wp\" (UID: \"b995d863-4e5c-487b-84a8-c09faf07bc36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp"
Jul 20 19:30:11.375023 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.374853    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b995d863-4e5c-487b-84a8-c09faf07bc36-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vb2wp\" (UID: \"b995d863-4e5c-487b-84a8-c09faf07bc36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp"
Jul 20 19:30:11.475654 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.475615    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh2g9\" (UniqueName: \"kubernetes.io/projected/b995d863-4e5c-487b-84a8-c09faf07bc36-kube-api-access-wh2g9\") pod \"cert-manager-operator-controller-manager-55c444bf86-vb2wp\" (UID: \"b995d863-4e5c-487b-84a8-c09faf07bc36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp"
Jul 20 19:30:11.475654 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.475660    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b995d863-4e5c-487b-84a8-c09faf07bc36-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vb2wp\" (UID: \"b995d863-4e5c-487b-84a8-c09faf07bc36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp"
Jul 20 19:30:11.476097 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.476078    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b995d863-4e5c-487b-84a8-c09faf07bc36-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-vb2wp\" (UID: \"b995d863-4e5c-487b-84a8-c09faf07bc36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp"
Jul 20 19:30:11.485085 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.485062    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh2g9\" (UniqueName: \"kubernetes.io/projected/b995d863-4e5c-487b-84a8-c09faf07bc36-kube-api-access-wh2g9\") pod \"cert-manager-operator-controller-manager-55c444bf86-vb2wp\" (UID: \"b995d863-4e5c-487b-84a8-c09faf07bc36\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp"
Jul 20 19:30:11.600347 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.600233    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp"
Jul 20 19:30:11.738308 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:11.738272    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp"]
Jul 20 19:30:11.742111 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:30:11.742074    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb995d863_4e5c_487b_84a8_c09faf07bc36.slice/crio-857a4d0b13c5947bd957cb5c41ef577326096bcdce73e3d8248548c9535696bc WatchSource:0}: Error finding container 857a4d0b13c5947bd957cb5c41ef577326096bcdce73e3d8248548c9535696bc: Status 404 returned error can't find the container with id 857a4d0b13c5947bd957cb5c41ef577326096bcdce73e3d8248548c9535696bc
Jul 20 19:30:12.116949 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:12.116916    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp" event={"ID":"b995d863-4e5c-487b-84a8-c09faf07bc36","Type":"ContainerStarted","Data":"857a4d0b13c5947bd957cb5c41ef577326096bcdce73e3d8248548c9535696bc"}
Jul 20 19:30:16.133229 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:16.133196    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp" event={"ID":"b995d863-4e5c-487b-84a8-c09faf07bc36","Type":"ContainerStarted","Data":"c97fc43e741dcac47f2fafed482bf5aeebdb4bcc1fea658e31b5b803adcbaba7"}
Jul 20 19:30:16.176176 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:16.176109    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-vb2wp" podStartSLOduration=1.67577159 podStartE2EDuration="5.176087677s" totalImagesPullingTime="3.500316087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:30:16.174196216 +0000 UTC m=+468.260147214" watchObservedRunningTime="2026-07-20 19:30:16.176087677 +0000 UTC m=+468.262038675"
Jul 20 19:30:26.564632 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.564601    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-25tft"]
Jul 20 19:30:26.567430 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.567413    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-25tft"
Jul 20 19:30:26.569797 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.569775    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 20 19:30:26.570508 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.570489    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-fvrhs\""
Jul 20 19:30:26.570635 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.570509    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 20 19:30:26.581715 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.581687    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-25tft"]
Jul 20 19:30:26.683068 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.683029    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkd4x\" (UniqueName: \"kubernetes.io/projected/615a72d2-46a6-4e5d-8d3d-7325accd4e87-kube-api-access-zkd4x\") pod \"cert-manager-55b4d954d5-25tft\" (UID: \"615a72d2-46a6-4e5d-8d3d-7325accd4e87\") " pod="cert-manager/cert-manager-55b4d954d5-25tft"
Jul 20 19:30:26.683324 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.683091    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/615a72d2-46a6-4e5d-8d3d-7325accd4e87-bound-sa-token\") pod \"cert-manager-55b4d954d5-25tft\" (UID: \"615a72d2-46a6-4e5d-8d3d-7325accd4e87\") " pod="cert-manager/cert-manager-55b4d954d5-25tft"
Jul 20 19:30:26.784303 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.784244    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/615a72d2-46a6-4e5d-8d3d-7325accd4e87-bound-sa-token\") pod \"cert-manager-55b4d954d5-25tft\" (UID: \"615a72d2-46a6-4e5d-8d3d-7325accd4e87\") " pod="cert-manager/cert-manager-55b4d954d5-25tft"
Jul 20 19:30:26.784450 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.784349    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkd4x\" (UniqueName: \"kubernetes.io/projected/615a72d2-46a6-4e5d-8d3d-7325accd4e87-kube-api-access-zkd4x\") pod \"cert-manager-55b4d954d5-25tft\" (UID: \"615a72d2-46a6-4e5d-8d3d-7325accd4e87\") " pod="cert-manager/cert-manager-55b4d954d5-25tft"
Jul 20 19:30:26.792815 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.792777    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/615a72d2-46a6-4e5d-8d3d-7325accd4e87-bound-sa-token\") pod \"cert-manager-55b4d954d5-25tft\" (UID: \"615a72d2-46a6-4e5d-8d3d-7325accd4e87\") " pod="cert-manager/cert-manager-55b4d954d5-25tft"
Jul 20 19:30:26.792944 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.792886    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkd4x\" (UniqueName: \"kubernetes.io/projected/615a72d2-46a6-4e5d-8d3d-7325accd4e87-kube-api-access-zkd4x\") pod \"cert-manager-55b4d954d5-25tft\" (UID: \"615a72d2-46a6-4e5d-8d3d-7325accd4e87\") " pod="cert-manager/cert-manager-55b4d954d5-25tft"
Jul 20 19:30:26.876156 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:26.876124    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-25tft"
Jul 20 19:30:27.004569 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:27.004536    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-25tft"]
Jul 20 19:30:27.007120 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:30:27.007083    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod615a72d2_46a6_4e5d_8d3d_7325accd4e87.slice/crio-cb6dd2d26f64438909aa06ec2341dd085bbadf6f8bb4665cf4f65763d73660f8 WatchSource:0}: Error finding container cb6dd2d26f64438909aa06ec2341dd085bbadf6f8bb4665cf4f65763d73660f8: Status 404 returned error can't find the container with id cb6dd2d26f64438909aa06ec2341dd085bbadf6f8bb4665cf4f65763d73660f8
Jul 20 19:30:27.171137 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:27.171048    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-25tft" event={"ID":"615a72d2-46a6-4e5d-8d3d-7325accd4e87","Type":"ContainerStarted","Data":"cb6dd2d26f64438909aa06ec2341dd085bbadf6f8bb4665cf4f65763d73660f8"}
Jul 20 19:30:31.186083 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:31.186044    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-25tft" event={"ID":"615a72d2-46a6-4e5d-8d3d-7325accd4e87","Type":"ContainerStarted","Data":"f4250c8d5697763d8ae9323333066b703da9c4b73cf5bd7e469362554deba430"}
Jul 20 19:30:31.219319 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:31.219248    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-25tft" podStartSLOduration=2.006705028 podStartE2EDuration="5.219232688s" totalImagesPullingTime="3.21252766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:30:31.218410625 +0000 UTC m=+483.304361638" watchObservedRunningTime="2026-07-20 19:30:31.219232688 +0000 UTC m=+483.305183685"
Jul 20 19:30:47.959548 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:47.959506    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"]
Jul 20 19:30:47.990532 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:47.990501    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"]
Jul 20 19:30:47.990687 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:47.990601    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:47.994705 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:47.994676    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 20 19:30:47.994705 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:47.994699    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 20 19:30:47.994904 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:47.994710    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 20 19:30:47.994904 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:47.994712    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2kwsn\""
Jul 20 19:30:47.994904 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:47.994698    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 20 19:30:48.150502 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.150462    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/76d56700-1f80-437b-9565-9a3d46e7f3d9-manifests\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.150502 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.150504    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76d56700-1f80-437b-9565-9a3d46e7f3d9-tmp\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.150737 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.150540    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/76d56700-1f80-437b-9565-9a3d46e7f3d9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.150737 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.150566    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76d56700-1f80-437b-9565-9a3d46e7f3d9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.150737 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.150605    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6rkz\" (UniqueName: \"kubernetes.io/projected/76d56700-1f80-437b-9565-9a3d46e7f3d9-kube-api-access-z6rkz\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.252046 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.251966    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/76d56700-1f80-437b-9565-9a3d46e7f3d9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.252046 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.251999    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76d56700-1f80-437b-9565-9a3d46e7f3d9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.252046 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.252015    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6rkz\" (UniqueName: \"kubernetes.io/projected/76d56700-1f80-437b-9565-9a3d46e7f3d9-kube-api-access-z6rkz\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.252340 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.252067    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/76d56700-1f80-437b-9565-9a3d46e7f3d9-manifests\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.252340 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.252086    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76d56700-1f80-437b-9565-9a3d46e7f3d9-tmp\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.252547 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.252521    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/76d56700-1f80-437b-9565-9a3d46e7f3d9-manifests\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.254356 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.254337    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76d56700-1f80-437b-9565-9a3d46e7f3d9-tmp\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.254463 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.254422    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76d56700-1f80-437b-9565-9a3d46e7f3d9-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.254627 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.254610    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/76d56700-1f80-437b-9565-9a3d46e7f3d9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.259868 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.259847    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6rkz\" (UniqueName: \"kubernetes.io/projected/76d56700-1f80-437b-9565-9a3d46e7f3d9-kube-api-access-z6rkz\") pod \"opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr\" (UID: \"76d56700-1f80-437b-9565-9a3d46e7f3d9\") " pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.300141 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.300110    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:48.463857 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:48.463702    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"]
Jul 20 19:30:48.466578 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:30:48.466547    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d56700_1f80_437b_9565_9a3d46e7f3d9.slice/crio-0639c7c4cefca48dc5b8044a2f82fecfd343c2e8dd12e2313a3afe793d5a72fa WatchSource:0}: Error finding container 0639c7c4cefca48dc5b8044a2f82fecfd343c2e8dd12e2313a3afe793d5a72fa: Status 404 returned error can't find the container with id 0639c7c4cefca48dc5b8044a2f82fecfd343c2e8dd12e2313a3afe793d5a72fa
Jul 20 19:30:49.250918 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:49.250876    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr" event={"ID":"76d56700-1f80-437b-9565-9a3d46e7f3d9","Type":"ContainerStarted","Data":"0639c7c4cefca48dc5b8044a2f82fecfd343c2e8dd12e2313a3afe793d5a72fa"}
Jul 20 19:30:52.263898 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:52.263855    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr" event={"ID":"76d56700-1f80-437b-9565-9a3d46e7f3d9","Type":"ContainerStarted","Data":"ddd2828d40cf5dc645a154f219c1b167b4ce609a66df5365eb76660ca4cbb443"}
Jul 20 19:30:53.269212 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:53.269177    2550 generic.go:356] "Generic (PLEG): container finished" podID="76d56700-1f80-437b-9565-9a3d46e7f3d9" containerID="ddd2828d40cf5dc645a154f219c1b167b4ce609a66df5365eb76660ca4cbb443" exitCode=0
Jul 20 19:30:53.269604 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:53.269244    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr" event={"ID":"76d56700-1f80-437b-9565-9a3d46e7f3d9","Type":"ContainerDied","Data":"ddd2828d40cf5dc645a154f219c1b167b4ce609a66df5365eb76660ca4cbb443"}
Jul 20 19:30:54.274764 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:54.274730    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr" event={"ID":"76d56700-1f80-437b-9565-9a3d46e7f3d9","Type":"ContainerStarted","Data":"6e5358ab76de658c0b8f5103168933c5db685de7e827816c6979e97a9334285f"}
Jul 20 19:30:54.275143 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:54.274855    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:30:54.295717 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:30:54.295661    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr" podStartSLOduration=2.862025406 podStartE2EDuration="7.295644992s" totalImagesPullingTime="3.37087005s" totalInitContainerRuntime="1.062749536s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:30:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:30:54.294496326 +0000 UTC m=+506.380447325" watchObservedRunningTime="2026-07-20 19:30:54.295644992 +0000 UTC m=+506.381596029"
Jul 20 19:31:05.281503 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:05.281472    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr"
Jul 20 19:31:07.552798 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.552763    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"]
Jul 20 19:31:07.557781 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.557759    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"
Jul 20 19:31:07.561780 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.561757    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 20 19:31:07.561898 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.561779    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 20 19:31:07.561898 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.561784    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 20 19:31:07.561898 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.561760    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 20 19:31:07.569783 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.569761    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"]
Jul 20 19:31:07.701016 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.700986    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45994ef8-30f2-48c1-8707-44b700e57a2b-tls-certs\") pod \"kube-auth-proxy-7578c486d7-vrkgc\" (UID: \"45994ef8-30f2-48c1-8707-44b700e57a2b\") " pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"
Jul 20 19:31:07.701173 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.701030    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45994ef8-30f2-48c1-8707-44b700e57a2b-tmp\") pod \"kube-auth-proxy-7578c486d7-vrkgc\" (UID: \"45994ef8-30f2-48c1-8707-44b700e57a2b\") " pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"
Jul 20 19:31:07.701173 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.701059    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxfsz\" (UniqueName: \"kubernetes.io/projected/45994ef8-30f2-48c1-8707-44b700e57a2b-kube-api-access-qxfsz\") pod \"kube-auth-proxy-7578c486d7-vrkgc\" (UID: \"45994ef8-30f2-48c1-8707-44b700e57a2b\") " pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"
Jul 20 19:31:07.769605 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.769574    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n7b9c"]
Jul 20 19:31:07.771684 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.771669    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c"
Jul 20 19:31:07.774484 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.774461    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 20 19:31:07.774572 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.774475    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-tfvp7\""
Jul 20 19:31:07.786036 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.786010    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n7b9c"]
Jul 20 19:31:07.802549 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.802485    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45994ef8-30f2-48c1-8707-44b700e57a2b-tls-certs\") pod \"kube-auth-proxy-7578c486d7-vrkgc\" (UID: \"45994ef8-30f2-48c1-8707-44b700e57a2b\") " pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"
Jul 20 19:31:07.802717 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.802604    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45994ef8-30f2-48c1-8707-44b700e57a2b-tmp\") pod \"kube-auth-proxy-7578c486d7-vrkgc\" (UID: \"45994ef8-30f2-48c1-8707-44b700e57a2b\") " pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"
Jul 20 19:31:07.802717 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.802661    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxfsz\" (UniqueName: \"kubernetes.io/projected/45994ef8-30f2-48c1-8707-44b700e57a2b-kube-api-access-qxfsz\") pod \"kube-auth-proxy-7578c486d7-vrkgc\" (UID: \"45994ef8-30f2-48c1-8707-44b700e57a2b\") " pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"
Jul 20 19:31:07.806484 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.805572    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45994ef8-30f2-48c1-8707-44b700e57a2b-tls-certs\") pod \"kube-auth-proxy-7578c486d7-vrkgc\" (UID: \"45994ef8-30f2-48c1-8707-44b700e57a2b\") " pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"
Jul 20 19:31:07.806484 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.805616    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/45994ef8-30f2-48c1-8707-44b700e57a2b-tmp\") pod \"kube-auth-proxy-7578c486d7-vrkgc\" (UID: \"45994ef8-30f2-48c1-8707-44b700e57a2b\") " pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"
Jul 20 19:31:07.818498 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.818467    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxfsz\" (UniqueName: \"kubernetes.io/projected/45994ef8-30f2-48c1-8707-44b700e57a2b-kube-api-access-qxfsz\") pod \"kube-auth-proxy-7578c486d7-vrkgc\" (UID: \"45994ef8-30f2-48c1-8707-44b700e57a2b\") " pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"
Jul 20 19:31:07.867172 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.867136    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"
Jul 20 19:31:07.903113 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.903078    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp8l8\" (UniqueName: \"kubernetes.io/projected/13b6218d-ee95-4de8-8171-26749fc7da1d-kube-api-access-gp8l8\") pod \"odh-model-controller-7998c67dc4-n7b9c\" (UID: \"13b6218d-ee95-4de8-8171-26749fc7da1d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c"
Jul 20 19:31:07.903294 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.903137    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13b6218d-ee95-4de8-8171-26749fc7da1d-cert\") pod \"odh-model-controller-7998c67dc4-n7b9c\" (UID: \"13b6218d-ee95-4de8-8171-26749fc7da1d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c"
Jul 20 19:31:07.994576 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:07.994414    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc"]
Jul 20 19:31:07.997118 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:31:07.997087    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45994ef8_30f2_48c1_8707_44b700e57a2b.slice/crio-4bf5ef7407b7b12227e489d553c4f0e733f31a117b7a1f32a54923ec8170378d WatchSource:0}: Error finding container 4bf5ef7407b7b12227e489d553c4f0e733f31a117b7a1f32a54923ec8170378d: Status 404 returned error can't find the container with id 4bf5ef7407b7b12227e489d553c4f0e733f31a117b7a1f32a54923ec8170378d
Jul 20 19:31:08.004204 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.004180    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp8l8\" (UniqueName: \"kubernetes.io/projected/13b6218d-ee95-4de8-8171-26749fc7da1d-kube-api-access-gp8l8\") pod \"odh-model-controller-7998c67dc4-n7b9c\" (UID: \"13b6218d-ee95-4de8-8171-26749fc7da1d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c"
Jul 20 19:31:08.004309 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.004223    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13b6218d-ee95-4de8-8171-26749fc7da1d-cert\") pod \"odh-model-controller-7998c67dc4-n7b9c\" (UID: \"13b6218d-ee95-4de8-8171-26749fc7da1d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c"
Jul 20 19:31:08.006641 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.006619    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13b6218d-ee95-4de8-8171-26749fc7da1d-cert\") pod \"odh-model-controller-7998c67dc4-n7b9c\" (UID: \"13b6218d-ee95-4de8-8171-26749fc7da1d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c"
Jul 20 19:31:08.012385 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.012366    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp8l8\" (UniqueName: \"kubernetes.io/projected/13b6218d-ee95-4de8-8171-26749fc7da1d-kube-api-access-gp8l8\") pod \"odh-model-controller-7998c67dc4-n7b9c\" (UID: \"13b6218d-ee95-4de8-8171-26749fc7da1d\") " pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c"
Jul 20 19:31:08.081194 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.081106    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c"
Jul 20 19:31:08.218808 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.218776    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n7b9c"]
Jul 20 19:31:08.221900 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:31:08.221873    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b6218d_ee95_4de8_8171_26749fc7da1d.slice/crio-80e878229ee9111d8306b5718ce3ee1b997f648866818ca6c1158cfa29c4feab WatchSource:0}: Error finding container 80e878229ee9111d8306b5718ce3ee1b997f648866818ca6c1158cfa29c4feab: Status 404 returned error can't find the container with id 80e878229ee9111d8306b5718ce3ee1b997f648866818ca6c1158cfa29c4feab
Jul 20 19:31:08.327449 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.327417    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"]
Jul 20 19:31:08.330225 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.330191    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c" event={"ID":"13b6218d-ee95-4de8-8171-26749fc7da1d","Type":"ContainerStarted","Data":"80e878229ee9111d8306b5718ce3ee1b997f648866818ca6c1158cfa29c4feab"}
Jul 20 19:31:08.330358 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.330307    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.330620 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.330587    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc" event={"ID":"45994ef8-30f2-48c1-8707-44b700e57a2b","Type":"ContainerStarted","Data":"4bf5ef7407b7b12227e489d553c4f0e733f31a117b7a1f32a54923ec8170378d"}
Jul 20 19:31:08.332777 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.332716    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 20 19:31:08.332885 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.332813    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bz5bb\""
Jul 20 19:31:08.332885 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.332817    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 20 19:31:08.333095 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.333078    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 20 19:31:08.333179 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.333099    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 20 19:31:08.333383 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.333367    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 20 19:31:08.341298 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.341278    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"]
Jul 20 19:31:08.507748 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.507720    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad9d7768-e440-4657-8d4e-b575372f7323-metrics-cert\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.507940 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.507765    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9f5n\" (UniqueName: \"kubernetes.io/projected/ad9d7768-e440-4657-8d4e-b575372f7323-kube-api-access-k9f5n\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.507940 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.507836    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad9d7768-e440-4657-8d4e-b575372f7323-manager-config\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.507940 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.507868    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad9d7768-e440-4657-8d4e-b575372f7323-cert\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.609195 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.609107    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9f5n\" (UniqueName: \"kubernetes.io/projected/ad9d7768-e440-4657-8d4e-b575372f7323-kube-api-access-k9f5n\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.609195 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.609171    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad9d7768-e440-4657-8d4e-b575372f7323-manager-config\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.609725 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.609199    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad9d7768-e440-4657-8d4e-b575372f7323-cert\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.609725 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.609227    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad9d7768-e440-4657-8d4e-b575372f7323-metrics-cert\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.610021 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.609998    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad9d7768-e440-4657-8d4e-b575372f7323-manager-config\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.611824 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.611798    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad9d7768-e440-4657-8d4e-b575372f7323-metrics-cert\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.611933 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.611798    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad9d7768-e440-4657-8d4e-b575372f7323-cert\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.620150 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.620128    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9f5n\" (UniqueName: \"kubernetes.io/projected/ad9d7768-e440-4657-8d4e-b575372f7323-kube-api-access-k9f5n\") pod \"lws-controller-manager-7c66c54875-9j685\" (UID: \"ad9d7768-e440-4657-8d4e-b575372f7323\") " pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.639628 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.639605    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:08.802143 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:08.802094    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"]
Jul 20 19:31:09.337921 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:09.337880    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685" event={"ID":"ad9d7768-e440-4657-8d4e-b575372f7323","Type":"ContainerStarted","Data":"ac5ec26a741517515db58d44efbcf8b962b63856b771f7a782d987806c06a0d4"}
Jul 20 19:31:13.361058 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:13.361020    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c" event={"ID":"13b6218d-ee95-4de8-8171-26749fc7da1d","Type":"ContainerStarted","Data":"c915df6c0e03190c76c93755515f68b541238eaa756cc046022fb96b255d483f"}
Jul 20 19:31:13.361526 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:13.361148    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c"
Jul 20 19:31:13.362601 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:13.362573    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc" event={"ID":"45994ef8-30f2-48c1-8707-44b700e57a2b","Type":"ContainerStarted","Data":"0cfed233639595f60d9c021bdcdf3d7991f906593a04e3b49634991f2a57902e"}
Jul 20 19:31:13.363878 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:13.363855    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685" event={"ID":"ad9d7768-e440-4657-8d4e-b575372f7323","Type":"ContainerStarted","Data":"bdbc5850198dce60a8f9dbcc431aebff1177e32d9e5ab21cde85b924f2643800"}
Jul 20 19:31:13.363988 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:13.363954    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:31:13.376984 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:13.376943    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c" podStartSLOduration=1.8252129259999998 podStartE2EDuration="6.37693169s" totalImagesPullingTime="4.551718764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:31:13.376029868 +0000 UTC m=+525.461980868" watchObservedRunningTime="2026-07-20 19:31:13.37693169 +0000 UTC m=+525.462882727"
Jul 20 19:31:13.390844 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:13.390796    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685" podStartSLOduration=1.424893734 podStartE2EDuration="5.390781756s" totalImagesPullingTime="3.965888022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:31:13.390780745 +0000 UTC m=+525.476731744" watchObservedRunningTime="2026-07-20 19:31:13.390781756 +0000 UTC m=+525.476732753"
Jul 20 19:31:13.405943 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:13.405901    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7578c486d7-vrkgc" podStartSLOduration=1.637094159 podStartE2EDuration="6.405886898s" totalImagesPullingTime="4.768792739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:31:13.404609349 +0000 UTC m=+525.490560349" watchObservedRunningTime="2026-07-20 19:31:13.405886898 +0000 UTC m=+525.491837895"
Jul 20 19:31:24.370323 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:24.370228    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-n7b9c"
Jul 20 19:31:24.370904 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:31:24.370337    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c66c54875-9j685"
Jul 20 19:32:28.352872 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:28.352846    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:32:28.353449 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:28.352942    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:32:52.192277 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.192189    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"]
Jul 20 19:32:52.194235 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.194217    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:52.196461 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.196440    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 20 19:32:52.196461 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.196449    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 20 19:32:52.197017 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.196998    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 20 19:32:52.197126 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.197017    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 20 19:32:52.197126 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.197031    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gp7gx\""
Jul 20 19:32:52.210417 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.210393    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"]
Jul 20 19:32:52.294760 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.294720    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6rmm\" (UniqueName: \"kubernetes.io/projected/53d4d1e3-1196-4995-a421-44c48295b555-kube-api-access-v6rmm\") pod \"kuadrant-console-plugin-66b5774985-9nzjq\" (UID: \"53d4d1e3-1196-4995-a421-44c48295b555\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:52.294760 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.294766    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/53d4d1e3-1196-4995-a421-44c48295b555-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9nzjq\" (UID: \"53d4d1e3-1196-4995-a421-44c48295b555\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:52.294984 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.294870    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53d4d1e3-1196-4995-a421-44c48295b555-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9nzjq\" (UID: \"53d4d1e3-1196-4995-a421-44c48295b555\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:52.395246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.395196    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53d4d1e3-1196-4995-a421-44c48295b555-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9nzjq\" (UID: \"53d4d1e3-1196-4995-a421-44c48295b555\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:52.395246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.395249    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6rmm\" (UniqueName: \"kubernetes.io/projected/53d4d1e3-1196-4995-a421-44c48295b555-kube-api-access-v6rmm\") pod \"kuadrant-console-plugin-66b5774985-9nzjq\" (UID: \"53d4d1e3-1196-4995-a421-44c48295b555\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:52.395524 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.395318    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/53d4d1e3-1196-4995-a421-44c48295b555-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9nzjq\" (UID: \"53d4d1e3-1196-4995-a421-44c48295b555\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:52.395524 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:32:52.395436    2550 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 20 19:32:52.395524 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:32:52.395516    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d4d1e3-1196-4995-a421-44c48295b555-plugin-serving-cert podName:53d4d1e3-1196-4995-a421-44c48295b555 nodeName:}" failed. No retries permitted until 2026-07-20 19:32:52.895495419 +0000 UTC m=+624.981446401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/53d4d1e3-1196-4995-a421-44c48295b555-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-9nzjq" (UID: "53d4d1e3-1196-4995-a421-44c48295b555") : secret "plugin-serving-cert" not found
Jul 20 19:32:52.395884 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.395862    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/53d4d1e3-1196-4995-a421-44c48295b555-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9nzjq\" (UID: \"53d4d1e3-1196-4995-a421-44c48295b555\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:52.405247 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.405224    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6rmm\" (UniqueName: \"kubernetes.io/projected/53d4d1e3-1196-4995-a421-44c48295b555-kube-api-access-v6rmm\") pod \"kuadrant-console-plugin-66b5774985-9nzjq\" (UID: \"53d4d1e3-1196-4995-a421-44c48295b555\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:52.899137 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.899097    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/53d4d1e3-1196-4995-a421-44c48295b555-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9nzjq\" (UID: \"53d4d1e3-1196-4995-a421-44c48295b555\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:52.901608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:52.901585    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/53d4d1e3-1196-4995-a421-44c48295b555-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9nzjq\" (UID: \"53d4d1e3-1196-4995-a421-44c48295b555\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:53.103054 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:53.103011    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"
Jul 20 19:32:53.232939 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:53.232786    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq"]
Jul 20 19:32:53.235447 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:32:53.235414    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53d4d1e3_1196_4995_a421_44c48295b555.slice/crio-6d7ff9cbd1c9d9d50437226f917102a36593ec71ec6d12b453010d06c4a0ac6f WatchSource:0}: Error finding container 6d7ff9cbd1c9d9d50437226f917102a36593ec71ec6d12b453010d06c4a0ac6f: Status 404 returned error can't find the container with id 6d7ff9cbd1c9d9d50437226f917102a36593ec71ec6d12b453010d06c4a0ac6f
Jul 20 19:32:53.755614 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:32:53.755580    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq" event={"ID":"53d4d1e3-1196-4995-a421-44c48295b555","Type":"ContainerStarted","Data":"6d7ff9cbd1c9d9d50437226f917102a36593ec71ec6d12b453010d06c4a0ac6f"}
Jul 20 19:33:16.863497 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:16.863401    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq" event={"ID":"53d4d1e3-1196-4995-a421-44c48295b555","Type":"ContainerStarted","Data":"a6dcf27fb2be53c8f7650903be24c01e7d1364bbe5322bf11d9a9fb2ffe74428"}
Jul 20 19:33:16.879206 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:16.879160    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9nzjq" podStartSLOduration=1.634954967 podStartE2EDuration="24.879146445s" totalImagesPullingTime="23.244191478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:33:16.878014218 +0000 UTC m=+648.963965215" watchObservedRunningTime="2026-07-20 19:33:16.879146445 +0000 UTC m=+648.965097443"
Jul 20 19:33:40.773932 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:40.773894    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-schlm"]
Jul 20 19:33:41.400052 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.400005    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-schlm"]
Jul 20 19:33:41.400052 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.400057    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-7248g"]
Jul 20 19:33:41.400333 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.400162    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-schlm"
Jul 20 19:33:41.402358 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.402332    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-m24bb\""
Jul 20 19:33:41.504672 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.504633    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2qgp\" (UniqueName: \"kubernetes.io/projected/42f682db-25c4-49d9-bd0f-71f09842a801-kube-api-access-h2qgp\") pod \"authorino-58cd955675-schlm\" (UID: \"42f682db-25c4-49d9-bd0f-71f09842a801\") " pod="kuadrant-system/authorino-58cd955675-schlm"
Jul 20 19:33:41.566487 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.566451    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7248g"]
Jul 20 19:33:41.566649 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.566547    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7248g"
Jul 20 19:33:41.606035 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.605997    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2qgp\" (UniqueName: \"kubernetes.io/projected/42f682db-25c4-49d9-bd0f-71f09842a801-kube-api-access-h2qgp\") pod \"authorino-58cd955675-schlm\" (UID: \"42f682db-25c4-49d9-bd0f-71f09842a801\") " pod="kuadrant-system/authorino-58cd955675-schlm"
Jul 20 19:33:41.613526 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.613497    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2qgp\" (UniqueName: \"kubernetes.io/projected/42f682db-25c4-49d9-bd0f-71f09842a801-kube-api-access-h2qgp\") pod \"authorino-58cd955675-schlm\" (UID: \"42f682db-25c4-49d9-bd0f-71f09842a801\") " pod="kuadrant-system/authorino-58cd955675-schlm"
Jul 20 19:33:41.707325 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.707224    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vncqw\" (UniqueName: \"kubernetes.io/projected/9b06fc3d-c89b-48c6-a806-c0a4f665634e-kube-api-access-vncqw\") pod \"authorino-75d556bd77-7248g\" (UID: \"9b06fc3d-c89b-48c6-a806-c0a4f665634e\") " pod="kuadrant-system/authorino-75d556bd77-7248g"
Jul 20 19:33:41.709081 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.709055    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-schlm"
Jul 20 19:33:41.808374 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.808335    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncqw\" (UniqueName: \"kubernetes.io/projected/9b06fc3d-c89b-48c6-a806-c0a4f665634e-kube-api-access-vncqw\") pod \"authorino-75d556bd77-7248g\" (UID: \"9b06fc3d-c89b-48c6-a806-c0a4f665634e\") " pod="kuadrant-system/authorino-75d556bd77-7248g"
Jul 20 19:33:41.816297 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.816267    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vncqw\" (UniqueName: \"kubernetes.io/projected/9b06fc3d-c89b-48c6-a806-c0a4f665634e-kube-api-access-vncqw\") pod \"authorino-75d556bd77-7248g\" (UID: \"9b06fc3d-c89b-48c6-a806-c0a4f665634e\") " pod="kuadrant-system/authorino-75d556bd77-7248g"
Jul 20 19:33:41.875889 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.875662    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7248g"
Jul 20 19:33:41.904916 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.904749    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-schlm"]
Jul 20 19:33:41.907655 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:33:41.907630    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42f682db_25c4_49d9_bd0f_71f09842a801.slice/crio-63d93f33eba51f1b66a446fa9486f14cd0da20d96270c7cf204fcd94d3091557 WatchSource:0}: Error finding container 63d93f33eba51f1b66a446fa9486f14cd0da20d96270c7cf204fcd94d3091557: Status 404 returned error can't find the container with id 63d93f33eba51f1b66a446fa9486f14cd0da20d96270c7cf204fcd94d3091557
Jul 20 19:33:41.909644 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.909614    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 19:33:41.961561 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:41.961487    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-schlm" event={"ID":"42f682db-25c4-49d9-bd0f-71f09842a801","Type":"ContainerStarted","Data":"63d93f33eba51f1b66a446fa9486f14cd0da20d96270c7cf204fcd94d3091557"}
Jul 20 19:33:42.008498 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:42.008474    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7248g"]
Jul 20 19:33:42.011000 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:33:42.010976    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b06fc3d_c89b_48c6_a806_c0a4f665634e.slice/crio-9cb49f9ed65ae6517212c44e6e03d8b4560415bc87ca502602e16e0d605c9ddb WatchSource:0}: Error finding container 9cb49f9ed65ae6517212c44e6e03d8b4560415bc87ca502602e16e0d605c9ddb: Status 404 returned error can't find the container with id 9cb49f9ed65ae6517212c44e6e03d8b4560415bc87ca502602e16e0d605c9ddb
Jul 20 19:33:42.966547 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:42.966514    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7248g" event={"ID":"9b06fc3d-c89b-48c6-a806-c0a4f665634e","Type":"ContainerStarted","Data":"9cb49f9ed65ae6517212c44e6e03d8b4560415bc87ca502602e16e0d605c9ddb"}
Jul 20 19:33:46.992034 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:46.991994    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7248g" event={"ID":"9b06fc3d-c89b-48c6-a806-c0a4f665634e","Type":"ContainerStarted","Data":"b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952"}
Jul 20 19:33:46.993377 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:46.993355    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-schlm" event={"ID":"42f682db-25c4-49d9-bd0f-71f09842a801","Type":"ContainerStarted","Data":"bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d"}
Jul 20 19:33:47.005874 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:47.005829    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-7248g" podStartSLOduration=2.672689959 podStartE2EDuration="7.00581425s" totalImagesPullingTime="4.333124291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:33:47.004575993 +0000 UTC m=+679.090526992" watchObservedRunningTime="2026-07-20 19:33:47.00581425 +0000 UTC m=+679.091765249"
Jul 20 19:33:47.019828 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:47.019783    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-schlm" podStartSLOduration=2.574105447 podStartE2EDuration="7.019766974s" totalImagesPullingTime="4.445661527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:33:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:33:47.018956324 +0000 UTC m=+679.104907323" watchObservedRunningTime="2026-07-20 19:33:47.019766974 +0000 UTC m=+679.105718025"
Jul 20 19:33:47.047661 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:47.047630    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-schlm"]
Jul 20 19:33:49.000559 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:49.000519    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-schlm" podUID="42f682db-25c4-49d9-bd0f-71f09842a801" containerName="authorino" containerID="cri-o://bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d" gracePeriod=30
Jul 20 19:33:49.258818 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:49.258765    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-schlm"
Jul 20 19:33:49.273337 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:49.273312    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2qgp\" (UniqueName: \"kubernetes.io/projected/42f682db-25c4-49d9-bd0f-71f09842a801-kube-api-access-h2qgp\") pod \"42f682db-25c4-49d9-bd0f-71f09842a801\" (UID: \"42f682db-25c4-49d9-bd0f-71f09842a801\") "
Jul 20 19:33:49.275652 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:49.275622    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f682db-25c4-49d9-bd0f-71f09842a801-kube-api-access-h2qgp" (OuterVolumeSpecName: "kube-api-access-h2qgp") pod "42f682db-25c4-49d9-bd0f-71f09842a801" (UID: "42f682db-25c4-49d9-bd0f-71f09842a801"). InnerVolumeSpecName "kube-api-access-h2qgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:33:49.374077 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:49.374043    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2qgp\" (UniqueName: \"kubernetes.io/projected/42f682db-25c4-49d9-bd0f-71f09842a801-kube-api-access-h2qgp\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:33:50.005737 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:50.005705    2550 generic.go:356] "Generic (PLEG): container finished" podID="42f682db-25c4-49d9-bd0f-71f09842a801" containerID="bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d" exitCode=0
Jul 20 19:33:50.006165 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:50.005744    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-schlm"
Jul 20 19:33:50.006165 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:50.005765    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-schlm" event={"ID":"42f682db-25c4-49d9-bd0f-71f09842a801","Type":"ContainerDied","Data":"bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d"}
Jul 20 19:33:50.006165 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:50.005795    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-schlm" event={"ID":"42f682db-25c4-49d9-bd0f-71f09842a801","Type":"ContainerDied","Data":"63d93f33eba51f1b66a446fa9486f14cd0da20d96270c7cf204fcd94d3091557"}
Jul 20 19:33:50.006165 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:50.005812    2550 scope.go:117] "RemoveContainer" containerID="bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d"
Jul 20 19:33:50.017231 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:50.016915    2550 scope.go:117] "RemoveContainer" containerID="bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d"
Jul 20 19:33:50.017231 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:33:50.017203    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d\": container with ID starting with bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d not found: ID does not exist" containerID="bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d"
Jul 20 19:33:50.017386 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:50.017241    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d"} err="failed to get container status \"bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d\": rpc error: code = NotFound desc = could not find container \"bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d\": container with ID starting with bad2e98a04fba99c197c708cab1fe2e1b7f3481efc7c1440d2778f9c73cfea7d not found: ID does not exist"
Jul 20 19:33:50.029746 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:50.029725    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-schlm"]
Jul 20 19:33:50.034098 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:50.034077    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-schlm"]
Jul 20 19:33:50.450772 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:33:50.450736    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42f682db-25c4-49d9-bd0f-71f09842a801" path="/var/lib/kubelet/pods/42f682db-25c4-49d9-bd0f-71f09842a801/volumes"
Jul 20 19:34:08.685490 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.685459    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-l7m2j"]
Jul 20 19:34:08.685873 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.685731    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f682db-25c4-49d9-bd0f-71f09842a801" containerName="authorino"
Jul 20 19:34:08.685873 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.685743    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f682db-25c4-49d9-bd0f-71f09842a801" containerName="authorino"
Jul 20 19:34:08.685873 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.685792    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f682db-25c4-49d9-bd0f-71f09842a801" containerName="authorino"
Jul 20 19:34:08.688439 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.688423    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-l7m2j"
Jul 20 19:34:08.695249 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.695220    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-l7m2j"]
Jul 20 19:34:08.822340 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.822303    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpldj\" (UniqueName: \"kubernetes.io/projected/242a219d-e214-443b-b0c2-f4d27b75ead7-kube-api-access-cpldj\") pod \"authorino-5d4b67c446-l7m2j\" (UID: \"242a219d-e214-443b-b0c2-f4d27b75ead7\") " pod="kuadrant-system/authorino-5d4b67c446-l7m2j"
Jul 20 19:34:08.862233 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.862199    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-l7m2j"]
Jul 20 19:34:08.862442 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:34:08.862423    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-cpldj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-l7m2j" podUID="242a219d-e214-443b-b0c2-f4d27b75ead7"
Jul 20 19:34:08.880954 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.880928    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-6xpq6"]
Jul 20 19:34:08.883877 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.883862    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-6xpq6"
Jul 20 19:34:08.885909 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.885893    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 20 19:34:08.891105 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.891081    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-6xpq6"]
Jul 20 19:34:08.923027 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.922998    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpldj\" (UniqueName: \"kubernetes.io/projected/242a219d-e214-443b-b0c2-f4d27b75ead7-kube-api-access-cpldj\") pod \"authorino-5d4b67c446-l7m2j\" (UID: \"242a219d-e214-443b-b0c2-f4d27b75ead7\") " pod="kuadrant-system/authorino-5d4b67c446-l7m2j"
Jul 20 19:34:08.923164 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.923039    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69b4e3a2-3552-4646-b034-f600b529a288-tls-cert\") pod \"authorino-76877994-6xpq6\" (UID: \"69b4e3a2-3552-4646-b034-f600b529a288\") " pod="kuadrant-system/authorino-76877994-6xpq6"
Jul 20 19:34:08.923164 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.923086    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vjg\" (UniqueName: \"kubernetes.io/projected/69b4e3a2-3552-4646-b034-f600b529a288-kube-api-access-k5vjg\") pod \"authorino-76877994-6xpq6\" (UID: \"69b4e3a2-3552-4646-b034-f600b529a288\") " pod="kuadrant-system/authorino-76877994-6xpq6"
Jul 20 19:34:08.930880 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.930854    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpldj\" (UniqueName: \"kubernetes.io/projected/242a219d-e214-443b-b0c2-f4d27b75ead7-kube-api-access-cpldj\") pod \"authorino-5d4b67c446-l7m2j\" (UID: \"242a219d-e214-443b-b0c2-f4d27b75ead7\") " pod="kuadrant-system/authorino-5d4b67c446-l7m2j"
Jul 20 19:34:08.974711 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.974637    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-6xpq6"]
Jul 20 19:34:08.974869 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:34:08.974851    2550 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-k5vjg tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-6xpq6" podUID="69b4e3a2-3552-4646-b034-f600b529a288"
Jul 20 19:34:08.999176 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:08.999147    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-dd847567c-k9bbj"]
Jul 20 19:34:09.002420 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.002399    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dd847567c-k9bbj"
Jul 20 19:34:09.009725 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.009694    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dd847567c-k9bbj"]
Jul 20 19:34:09.024367 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.024342    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vjg\" (UniqueName: \"kubernetes.io/projected/69b4e3a2-3552-4646-b034-f600b529a288-kube-api-access-k5vjg\") pod \"authorino-76877994-6xpq6\" (UID: \"69b4e3a2-3552-4646-b034-f600b529a288\") " pod="kuadrant-system/authorino-76877994-6xpq6"
Jul 20 19:34:09.024488 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.024398    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69b4e3a2-3552-4646-b034-f600b529a288-tls-cert\") pod \"authorino-76877994-6xpq6\" (UID: \"69b4e3a2-3552-4646-b034-f600b529a288\") " pod="kuadrant-system/authorino-76877994-6xpq6"
Jul 20 19:34:09.026722 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.026704    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69b4e3a2-3552-4646-b034-f600b529a288-tls-cert\") pod \"authorino-76877994-6xpq6\" (UID: \"69b4e3a2-3552-4646-b034-f600b529a288\") " pod="kuadrant-system/authorino-76877994-6xpq6"
Jul 20 19:34:09.031999 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.031975    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vjg\" (UniqueName: \"kubernetes.io/projected/69b4e3a2-3552-4646-b034-f600b529a288-kube-api-access-k5vjg\") pod \"authorino-76877994-6xpq6\" (UID: \"69b4e3a2-3552-4646-b034-f600b529a288\") " pod="kuadrant-system/authorino-76877994-6xpq6"
Jul 20 19:34:09.081980 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.081953    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-l7m2j"
Jul 20 19:34:09.082128 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.081953    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-6xpq6"
Jul 20 19:34:09.089008 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.088983    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-l7m2j"
Jul 20 19:34:09.091487 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.091471    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-6xpq6"
Jul 20 19:34:09.124885 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.124854    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/496b8a76-ac44-4525-b4d3-8c7c32e409c0-tls-cert\") pod \"authorino-dd847567c-k9bbj\" (UID: \"496b8a76-ac44-4525-b4d3-8c7c32e409c0\") " pod="kuadrant-system/authorino-dd847567c-k9bbj"
Jul 20 19:34:09.125024 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.124895    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjd77\" (UniqueName: \"kubernetes.io/projected/496b8a76-ac44-4525-b4d3-8c7c32e409c0-kube-api-access-tjd77\") pod \"authorino-dd847567c-k9bbj\" (UID: \"496b8a76-ac44-4525-b4d3-8c7c32e409c0\") " pod="kuadrant-system/authorino-dd847567c-k9bbj"
Jul 20 19:34:09.226164 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.226078    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69b4e3a2-3552-4646-b034-f600b529a288-tls-cert\") pod \"69b4e3a2-3552-4646-b034-f600b529a288\" (UID: \"69b4e3a2-3552-4646-b034-f600b529a288\") "
Jul 20 19:34:09.226164 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.226134    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpldj\" (UniqueName: \"kubernetes.io/projected/242a219d-e214-443b-b0c2-f4d27b75ead7-kube-api-access-cpldj\") pod \"242a219d-e214-443b-b0c2-f4d27b75ead7\" (UID: \"242a219d-e214-443b-b0c2-f4d27b75ead7\") "
Jul 20 19:34:09.226383 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.226170    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5vjg\" (UniqueName: \"kubernetes.io/projected/69b4e3a2-3552-4646-b034-f600b529a288-kube-api-access-k5vjg\") pod \"69b4e3a2-3552-4646-b034-f600b529a288\" (UID: \"69b4e3a2-3552-4646-b034-f600b529a288\") "
Jul 20 19:34:09.226383 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.226366    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/496b8a76-ac44-4525-b4d3-8c7c32e409c0-tls-cert\") pod \"authorino-dd847567c-k9bbj\" (UID: \"496b8a76-ac44-4525-b4d3-8c7c32e409c0\") " pod="kuadrant-system/authorino-dd847567c-k9bbj"
Jul 20 19:34:09.226471 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.226418    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjd77\" (UniqueName: \"kubernetes.io/projected/496b8a76-ac44-4525-b4d3-8c7c32e409c0-kube-api-access-tjd77\") pod \"authorino-dd847567c-k9bbj\" (UID: \"496b8a76-ac44-4525-b4d3-8c7c32e409c0\") " pod="kuadrant-system/authorino-dd847567c-k9bbj"
Jul 20 19:34:09.228441 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.228405    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/242a219d-e214-443b-b0c2-f4d27b75ead7-kube-api-access-cpldj" (OuterVolumeSpecName: "kube-api-access-cpldj") pod "242a219d-e214-443b-b0c2-f4d27b75ead7" (UID: "242a219d-e214-443b-b0c2-f4d27b75ead7"). InnerVolumeSpecName "kube-api-access-cpldj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:34:09.228441 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.228425    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b4e3a2-3552-4646-b034-f600b529a288-kube-api-access-k5vjg" (OuterVolumeSpecName: "kube-api-access-k5vjg") pod "69b4e3a2-3552-4646-b034-f600b529a288" (UID: "69b4e3a2-3552-4646-b034-f600b529a288"). InnerVolumeSpecName "kube-api-access-k5vjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:34:09.228441 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.228434    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b4e3a2-3552-4646-b034-f600b529a288-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "69b4e3a2-3552-4646-b034-f600b529a288" (UID: "69b4e3a2-3552-4646-b034-f600b529a288"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:34:09.228812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.228795    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/496b8a76-ac44-4525-b4d3-8c7c32e409c0-tls-cert\") pod \"authorino-dd847567c-k9bbj\" (UID: \"496b8a76-ac44-4525-b4d3-8c7c32e409c0\") " pod="kuadrant-system/authorino-dd847567c-k9bbj"
Jul 20 19:34:09.233550 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.233421    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjd77\" (UniqueName: \"kubernetes.io/projected/496b8a76-ac44-4525-b4d3-8c7c32e409c0-kube-api-access-tjd77\") pod \"authorino-dd847567c-k9bbj\" (UID: \"496b8a76-ac44-4525-b4d3-8c7c32e409c0\") " pod="kuadrant-system/authorino-dd847567c-k9bbj"
Jul 20 19:34:09.311400 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.311366    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dd847567c-k9bbj"
Jul 20 19:34:09.327639 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.327616    2550 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/69b4e3a2-3552-4646-b034-f600b529a288-tls-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:34:09.327639 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.327640    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpldj\" (UniqueName: \"kubernetes.io/projected/242a219d-e214-443b-b0c2-f4d27b75ead7-kube-api-access-cpldj\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:34:09.327765 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.327650    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5vjg\" (UniqueName: \"kubernetes.io/projected/69b4e3a2-3552-4646-b034-f600b529a288-kube-api-access-k5vjg\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:34:09.436525 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:09.436501    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dd847567c-k9bbj"]
Jul 20 19:34:09.437834 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:34:09.437808    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod496b8a76_ac44_4525_b4d3_8c7c32e409c0.slice/crio-966e511797fc59bf7f3f18ec5e0d06d59c839fe34b40ef9b29c3d3ecfa1e4bd6 WatchSource:0}: Error finding container 966e511797fc59bf7f3f18ec5e0d06d59c839fe34b40ef9b29c3d3ecfa1e4bd6: Status 404 returned error can't find the container with id 966e511797fc59bf7f3f18ec5e0d06d59c839fe34b40ef9b29c3d3ecfa1e4bd6
Jul 20 19:34:10.087875 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.087782    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dd847567c-k9bbj" event={"ID":"496b8a76-ac44-4525-b4d3-8c7c32e409c0","Type":"ContainerStarted","Data":"f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636"}
Jul 20 19:34:10.087875 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.087832    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dd847567c-k9bbj" event={"ID":"496b8a76-ac44-4525-b4d3-8c7c32e409c0","Type":"ContainerStarted","Data":"966e511797fc59bf7f3f18ec5e0d06d59c839fe34b40ef9b29c3d3ecfa1e4bd6"}
Jul 20 19:34:10.088323 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.087979    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-6xpq6"
Jul 20 19:34:10.088323 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.088148    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-l7m2j"
Jul 20 19:34:10.102663 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.102624    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-dd847567c-k9bbj" podStartSLOduration=1.751992718 podStartE2EDuration="2.102610936s" totalImagesPullingTime="350.618218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:34:10.101825508 +0000 UTC m=+702.187776507" watchObservedRunningTime="2026-07-20 19:34:10.102610936 +0000 UTC m=+702.188561934"
Jul 20 19:34:10.123767 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.123737    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-l7m2j"]
Jul 20 19:34:10.130097 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.130069    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-l7m2j"]
Jul 20 19:34:10.133747 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.133723    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7248g"]
Jul 20 19:34:10.133908 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.133886    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-7248g" podUID="9b06fc3d-c89b-48c6-a806-c0a4f665634e" containerName="authorino" containerID="cri-o://b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952" gracePeriod=30
Jul 20 19:34:10.157039 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.157016    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-6xpq6"]
Jul 20 19:34:10.163010 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.162988    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-6xpq6"]
Jul 20 19:34:10.370888 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.370864    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7248g"
Jul 20 19:34:10.437513 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.437482    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vncqw\" (UniqueName: \"kubernetes.io/projected/9b06fc3d-c89b-48c6-a806-c0a4f665634e-kube-api-access-vncqw\") pod \"9b06fc3d-c89b-48c6-a806-c0a4f665634e\" (UID: \"9b06fc3d-c89b-48c6-a806-c0a4f665634e\") "
Jul 20 19:34:10.439449 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.439427    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b06fc3d-c89b-48c6-a806-c0a4f665634e-kube-api-access-vncqw" (OuterVolumeSpecName: "kube-api-access-vncqw") pod "9b06fc3d-c89b-48c6-a806-c0a4f665634e" (UID: "9b06fc3d-c89b-48c6-a806-c0a4f665634e"). InnerVolumeSpecName "kube-api-access-vncqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:34:10.450314 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.450286    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="242a219d-e214-443b-b0c2-f4d27b75ead7" path="/var/lib/kubelet/pods/242a219d-e214-443b-b0c2-f4d27b75ead7/volumes"
Jul 20 19:34:10.450545 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.450534    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69b4e3a2-3552-4646-b034-f600b529a288" path="/var/lib/kubelet/pods/69b4e3a2-3552-4646-b034-f600b529a288/volumes"
Jul 20 19:34:10.538196 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:10.538167    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vncqw\" (UniqueName: \"kubernetes.io/projected/9b06fc3d-c89b-48c6-a806-c0a4f665634e-kube-api-access-vncqw\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:34:11.093844 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:11.093809    2550 generic.go:356] "Generic (PLEG): container finished" podID="9b06fc3d-c89b-48c6-a806-c0a4f665634e" containerID="b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952" exitCode=0
Jul 20 19:34:11.094290 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:11.093854    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-7248g"
Jul 20 19:34:11.094290 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:11.093898    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7248g" event={"ID":"9b06fc3d-c89b-48c6-a806-c0a4f665634e","Type":"ContainerDied","Data":"b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952"}
Jul 20 19:34:11.094290 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:11.093937    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-7248g" event={"ID":"9b06fc3d-c89b-48c6-a806-c0a4f665634e","Type":"ContainerDied","Data":"9cb49f9ed65ae6517212c44e6e03d8b4560415bc87ca502602e16e0d605c9ddb"}
Jul 20 19:34:11.094290 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:11.093957    2550 scope.go:117] "RemoveContainer" containerID="b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952"
Jul 20 19:34:11.102446 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:11.102427    2550 scope.go:117] "RemoveContainer" containerID="b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952"
Jul 20 19:34:11.102698 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:34:11.102680    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952\": container with ID starting with b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952 not found: ID does not exist" containerID="b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952"
Jul 20 19:34:11.102738 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:11.102706    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952"} err="failed to get container status \"b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952\": rpc error: code = NotFound desc = could not find container \"b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952\": container with ID starting with b5b627b6736b0886e87b54d478bb55fc371a76bc7ff30e275eee8f1eb3c58952 not found: ID does not exist"
Jul 20 19:34:11.107220 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:11.107199    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7248g"]
Jul 20 19:34:11.110412 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:11.110393    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-7248g"]
Jul 20 19:34:12.450160 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:12.450125    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b06fc3d-c89b-48c6-a806-c0a4f665634e" path="/var/lib/kubelet/pods/9b06fc3d-c89b-48c6-a806-c0a4f665634e/volumes"
Jul 20 19:34:15.147814 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.147777    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56d7856db5-js796"]
Jul 20 19:34:15.148275 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.148081    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b06fc3d-c89b-48c6-a806-c0a4f665634e" containerName="authorino"
Jul 20 19:34:15.148275 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.148094    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b06fc3d-c89b-48c6-a806-c0a4f665634e" containerName="authorino"
Jul 20 19:34:15.148275 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.148146    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b06fc3d-c89b-48c6-a806-c0a4f665634e" containerName="authorino"
Jul 20 19:34:15.152209 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.152187    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:15.154506 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.154475    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 20 19:34:15.154610 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.154505    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tk6n8\""
Jul 20 19:34:15.154610 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.154508    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 20 19:34:15.162645 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.162623    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56d7856db5-js796"]
Jul 20 19:34:15.276337 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.276294    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjzc\" (UniqueName: \"kubernetes.io/projected/f9c802a7-833a-41ec-a427-8445d5dc61b6-kube-api-access-6kjzc\") pod \"maas-controller-56d7856db5-js796\" (UID: \"f9c802a7-833a-41ec-a427-8445d5dc61b6\") " pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:15.276492 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.276410    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c802a7-833a-41ec-a427-8445d5dc61b6-webhook-cert\") pod \"maas-controller-56d7856db5-js796\" (UID: \"f9c802a7-833a-41ec-a427-8445d5dc61b6\") " pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:15.303353 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.303320    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-cwstv"]
Jul 20 19:34:15.306372 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.306355    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:34:15.316827 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.316804    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-cwstv"]
Jul 20 19:34:15.377477 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.377444    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c802a7-833a-41ec-a427-8445d5dc61b6-webhook-cert\") pod \"maas-controller-56d7856db5-js796\" (UID: \"f9c802a7-833a-41ec-a427-8445d5dc61b6\") " pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:15.377626 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.377496    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjzc\" (UniqueName: \"kubernetes.io/projected/f9c802a7-833a-41ec-a427-8445d5dc61b6-kube-api-access-6kjzc\") pod \"maas-controller-56d7856db5-js796\" (UID: \"f9c802a7-833a-41ec-a427-8445d5dc61b6\") " pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:15.377626 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:34:15.377580    2550 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 20 19:34:15.377626 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.377606    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljkjc\" (UniqueName: \"kubernetes.io/projected/641b18fb-85f7-438a-a43b-2318992db0c9-kube-api-access-ljkjc\") pod \"maas-controller-75c5bf7b68-cwstv\" (UID: \"641b18fb-85f7-438a-a43b-2318992db0c9\") " pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:34:15.377749 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:34:15.377632    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c802a7-833a-41ec-a427-8445d5dc61b6-webhook-cert podName:f9c802a7-833a-41ec-a427-8445d5dc61b6 nodeName:}" failed. No retries permitted until 2026-07-20 19:34:15.877617628 +0000 UTC m=+707.963568607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f9c802a7-833a-41ec-a427-8445d5dc61b6-webhook-cert") pod "maas-controller-56d7856db5-js796" (UID: "f9c802a7-833a-41ec-a427-8445d5dc61b6") : secret "maas-controller-webhook-cert" not found
Jul 20 19:34:15.377749 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.377665    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/641b18fb-85f7-438a-a43b-2318992db0c9-webhook-cert\") pod \"maas-controller-75c5bf7b68-cwstv\" (UID: \"641b18fb-85f7-438a-a43b-2318992db0c9\") " pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:34:15.388969 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.388945    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjzc\" (UniqueName: \"kubernetes.io/projected/f9c802a7-833a-41ec-a427-8445d5dc61b6-kube-api-access-6kjzc\") pod \"maas-controller-56d7856db5-js796\" (UID: \"f9c802a7-833a-41ec-a427-8445d5dc61b6\") " pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:15.478309 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.478225    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljkjc\" (UniqueName: \"kubernetes.io/projected/641b18fb-85f7-438a-a43b-2318992db0c9-kube-api-access-ljkjc\") pod \"maas-controller-75c5bf7b68-cwstv\" (UID: \"641b18fb-85f7-438a-a43b-2318992db0c9\") " pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:34:15.478309 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.478269    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/641b18fb-85f7-438a-a43b-2318992db0c9-webhook-cert\") pod \"maas-controller-75c5bf7b68-cwstv\" (UID: \"641b18fb-85f7-438a-a43b-2318992db0c9\") " pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:34:15.480615 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.480597    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/641b18fb-85f7-438a-a43b-2318992db0c9-webhook-cert\") pod \"maas-controller-75c5bf7b68-cwstv\" (UID: \"641b18fb-85f7-438a-a43b-2318992db0c9\") " pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:34:15.485506 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.485483    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljkjc\" (UniqueName: \"kubernetes.io/projected/641b18fb-85f7-438a-a43b-2318992db0c9-kube-api-access-ljkjc\") pod \"maas-controller-75c5bf7b68-cwstv\" (UID: \"641b18fb-85f7-438a-a43b-2318992db0c9\") " pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:34:15.615601 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.615565    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:34:15.741759 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.741737    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-cwstv"]
Jul 20 19:34:15.743972 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:34:15.743940    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641b18fb_85f7_438a_a43b_2318992db0c9.slice/crio-3418050dae2be8bfe0e2b8e3e980a81fc77a5e29072f66355794e41dfde32011 WatchSource:0}: Error finding container 3418050dae2be8bfe0e2b8e3e980a81fc77a5e29072f66355794e41dfde32011: Status 404 returned error can't find the container with id 3418050dae2be8bfe0e2b8e3e980a81fc77a5e29072f66355794e41dfde32011
Jul 20 19:34:15.882345 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.882306    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c802a7-833a-41ec-a427-8445d5dc61b6-webhook-cert\") pod \"maas-controller-56d7856db5-js796\" (UID: \"f9c802a7-833a-41ec-a427-8445d5dc61b6\") " pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:15.884825 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:15.884798    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c802a7-833a-41ec-a427-8445d5dc61b6-webhook-cert\") pod \"maas-controller-56d7856db5-js796\" (UID: \"f9c802a7-833a-41ec-a427-8445d5dc61b6\") " pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:16.062900 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:16.062818    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:16.123186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:16.123155    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5bf7b68-cwstv" event={"ID":"641b18fb-85f7-438a-a43b-2318992db0c9","Type":"ContainerStarted","Data":"3418050dae2be8bfe0e2b8e3e980a81fc77a5e29072f66355794e41dfde32011"}
Jul 20 19:34:16.189622 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:16.189598    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56d7856db5-js796"]
Jul 20 19:34:16.191785 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:34:16.191764    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c802a7_833a_41ec_a427_8445d5dc61b6.slice/crio-908b8d58e4ebe3437d6681934c91d94b5f10df38d5365cd618071d90495d245f WatchSource:0}: Error finding container 908b8d58e4ebe3437d6681934c91d94b5f10df38d5365cd618071d90495d245f: Status 404 returned error can't find the container with id 908b8d58e4ebe3437d6681934c91d94b5f10df38d5365cd618071d90495d245f
Jul 20 19:34:17.130002 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:17.129943    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56d7856db5-js796" event={"ID":"f9c802a7-833a-41ec-a427-8445d5dc61b6","Type":"ContainerStarted","Data":"908b8d58e4ebe3437d6681934c91d94b5f10df38d5365cd618071d90495d245f"}
Jul 20 19:34:19.140398 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:19.140360    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5bf7b68-cwstv" event={"ID":"641b18fb-85f7-438a-a43b-2318992db0c9","Type":"ContainerStarted","Data":"032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c"}
Jul 20 19:34:19.140830 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:19.140471    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:34:19.142069 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:19.142044    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56d7856db5-js796" event={"ID":"f9c802a7-833a-41ec-a427-8445d5dc61b6","Type":"ContainerStarted","Data":"39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1"}
Jul 20 19:34:19.142182 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:19.142087    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:19.221673 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:19.221624    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56d7856db5-js796" podStartSLOduration=2.107348242 podStartE2EDuration="4.221606305s" totalImagesPullingTime="2.114258063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:34:19.220273351 +0000 UTC m=+711.306224343" watchObservedRunningTime="2026-07-20 19:34:19.221606305 +0000 UTC m=+711.307557303"
Jul 20 19:34:19.222449 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:19.222411    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75c5bf7b68-cwstv" podStartSLOduration=1.6693256779999999 podStartE2EDuration="4.222396978s" totalImagesPullingTime="2.5530713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:34:19.182749336 +0000 UTC m=+711.268700358" watchObservedRunningTime="2026-07-20 19:34:19.222396978 +0000 UTC m=+711.308347977"
Jul 20 19:34:30.149242 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:30.149212    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:30.149647 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:30.149306    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:34:30.212882 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:30.212843    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56d7856db5-js796"]
Jul 20 19:34:30.213068 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:30.213041    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-56d7856db5-js796" podUID="f9c802a7-833a-41ec-a427-8445d5dc61b6" containerName="manager" containerID="cri-o://39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1" gracePeriod=30
Jul 20 19:34:30.461645 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:30.461619    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:30.605996 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:30.605961    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kjzc\" (UniqueName: \"kubernetes.io/projected/f9c802a7-833a-41ec-a427-8445d5dc61b6-kube-api-access-6kjzc\") pod \"f9c802a7-833a-41ec-a427-8445d5dc61b6\" (UID: \"f9c802a7-833a-41ec-a427-8445d5dc61b6\") "
Jul 20 19:34:30.606190 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:30.606012    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c802a7-833a-41ec-a427-8445d5dc61b6-webhook-cert\") pod \"f9c802a7-833a-41ec-a427-8445d5dc61b6\" (UID: \"f9c802a7-833a-41ec-a427-8445d5dc61b6\") "
Jul 20 19:34:30.608171 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:30.608144    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9c802a7-833a-41ec-a427-8445d5dc61b6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f9c802a7-833a-41ec-a427-8445d5dc61b6" (UID: "f9c802a7-833a-41ec-a427-8445d5dc61b6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:34:30.608248 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:30.608183    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c802a7-833a-41ec-a427-8445d5dc61b6-kube-api-access-6kjzc" (OuterVolumeSpecName: "kube-api-access-6kjzc") pod "f9c802a7-833a-41ec-a427-8445d5dc61b6" (UID: "f9c802a7-833a-41ec-a427-8445d5dc61b6"). InnerVolumeSpecName "kube-api-access-6kjzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:34:30.707450 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:30.707347    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kjzc\" (UniqueName: \"kubernetes.io/projected/f9c802a7-833a-41ec-a427-8445d5dc61b6-kube-api-access-6kjzc\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:34:30.707450 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:30.707394    2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f9c802a7-833a-41ec-a427-8445d5dc61b6-webhook-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:34:31.193603 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:31.193562    2550 generic.go:356] "Generic (PLEG): container finished" podID="f9c802a7-833a-41ec-a427-8445d5dc61b6" containerID="39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1" exitCode=0
Jul 20 19:34:31.194002 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:31.193625    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56d7856db5-js796"
Jul 20 19:34:31.194002 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:31.193645    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56d7856db5-js796" event={"ID":"f9c802a7-833a-41ec-a427-8445d5dc61b6","Type":"ContainerDied","Data":"39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1"}
Jul 20 19:34:31.194002 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:31.193732    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56d7856db5-js796" event={"ID":"f9c802a7-833a-41ec-a427-8445d5dc61b6","Type":"ContainerDied","Data":"908b8d58e4ebe3437d6681934c91d94b5f10df38d5365cd618071d90495d245f"}
Jul 20 19:34:31.194002 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:31.193757    2550 scope.go:117] "RemoveContainer" containerID="39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1"
Jul 20 19:34:31.202075 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:31.202058    2550 scope.go:117] "RemoveContainer" containerID="39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1"
Jul 20 19:34:31.202368 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:34:31.202348    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1\": container with ID starting with 39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1 not found: ID does not exist" containerID="39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1"
Jul 20 19:34:31.202433 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:31.202376    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1"} err="failed to get container status \"39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1\": rpc error: code = NotFound desc = could not find container \"39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1\": container with ID starting with 39d188670af8a6b0542e439d9699591e73ae4a36ce48eb20959cfc84f9b5a4a1 not found: ID does not exist"
Jul 20 19:34:31.213553 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:31.213531    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56d7856db5-js796"]
Jul 20 19:34:31.217670 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:31.217651    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56d7856db5-js796"]
Jul 20 19:34:32.449685 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:32.449656    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9c802a7-833a-41ec-a427-8445d5dc61b6" path="/var/lib/kubelet/pods/f9c802a7-833a-41ec-a427-8445d5dc61b6/volumes"
Jul 20 19:34:50.818356 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.818318    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"]
Jul 20 19:34:50.818866 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.818747    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c802a7-833a-41ec-a427-8445d5dc61b6" containerName="manager"
Jul 20 19:34:50.818866 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.818766    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c802a7-833a-41ec-a427-8445d5dc61b6" containerName="manager"
Jul 20 19:34:50.818866 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.818851    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c802a7-833a-41ec-a427-8445d5dc61b6" containerName="manager"
Jul 20 19:34:50.826737 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.826712    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:50.829863 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.829835    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-blbs7\""
Jul 20 19:34:50.829863 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.829860    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 20 19:34:50.830037 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.829840    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 20 19:34:50.830037 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.829885    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 20 19:34:50.834608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.834587    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"]
Jul 20 19:34:50.857473 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.857437    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56q8c\" (UniqueName: \"kubernetes.io/projected/524b1c09-1584-48ee-b2aa-bbd0bd65102c-kube-api-access-56q8c\") pod \"maas-api-9f859c8f8-vf9lt\" (UID: \"524b1c09-1584-48ee-b2aa-bbd0bd65102c\") " pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:50.857613 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.857494    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/524b1c09-1584-48ee-b2aa-bbd0bd65102c-maas-api-tls\") pod \"maas-api-9f859c8f8-vf9lt\" (UID: \"524b1c09-1584-48ee-b2aa-bbd0bd65102c\") " pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:50.857613 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.857608    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/524b1c09-1584-48ee-b2aa-bbd0bd65102c-openshift-service-ca\") pod \"maas-api-9f859c8f8-vf9lt\" (UID: \"524b1c09-1584-48ee-b2aa-bbd0bd65102c\") " pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:50.958721 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.958680    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56q8c\" (UniqueName: \"kubernetes.io/projected/524b1c09-1584-48ee-b2aa-bbd0bd65102c-kube-api-access-56q8c\") pod \"maas-api-9f859c8f8-vf9lt\" (UID: \"524b1c09-1584-48ee-b2aa-bbd0bd65102c\") " pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:50.958917 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.958737    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/524b1c09-1584-48ee-b2aa-bbd0bd65102c-maas-api-tls\") pod \"maas-api-9f859c8f8-vf9lt\" (UID: \"524b1c09-1584-48ee-b2aa-bbd0bd65102c\") " pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:50.958917 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.958793    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/524b1c09-1584-48ee-b2aa-bbd0bd65102c-openshift-service-ca\") pod \"maas-api-9f859c8f8-vf9lt\" (UID: \"524b1c09-1584-48ee-b2aa-bbd0bd65102c\") " pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:50.959465 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.959439    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/524b1c09-1584-48ee-b2aa-bbd0bd65102c-openshift-service-ca\") pod \"maas-api-9f859c8f8-vf9lt\" (UID: \"524b1c09-1584-48ee-b2aa-bbd0bd65102c\") " pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:50.961181 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.961161    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/524b1c09-1584-48ee-b2aa-bbd0bd65102c-maas-api-tls\") pod \"maas-api-9f859c8f8-vf9lt\" (UID: \"524b1c09-1584-48ee-b2aa-bbd0bd65102c\") " pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:50.966836 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:50.966813    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56q8c\" (UniqueName: \"kubernetes.io/projected/524b1c09-1584-48ee-b2aa-bbd0bd65102c-kube-api-access-56q8c\") pod \"maas-api-9f859c8f8-vf9lt\" (UID: \"524b1c09-1584-48ee-b2aa-bbd0bd65102c\") " pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:51.137409 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.137376    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:51.269941 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.269907    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"]
Jul 20 19:34:51.272521 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:34:51.272489    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524b1c09_1584_48ee_b2aa_bbd0bd65102c.slice/crio-b1dede92c230584750f97c1a87f139c993307b5b10605ae2dbdd92c5603f055b WatchSource:0}: Error finding container b1dede92c230584750f97c1a87f139c993307b5b10605ae2dbdd92c5603f055b: Status 404 returned error can't find the container with id b1dede92c230584750f97c1a87f139c993307b5b10605ae2dbdd92c5603f055b
Jul 20 19:34:51.312286 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.312239    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-4chbp"]
Jul 20 19:34:51.316425 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.316402    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-4chbp"
Jul 20 19:34:51.318879 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.318853    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 20 19:34:51.319009 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.318859    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jb4t8\""
Jul 20 19:34:51.332998 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.332974    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-4chbp"]
Jul 20 19:34:51.362808 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.362774    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19a7680-c1b3-49e7-8e02-e3d48ecba89d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-4chbp\" (UID: \"e19a7680-c1b3-49e7-8e02-e3d48ecba89d\") " pod="openshift-ingress/payload-processing-5bf47cd988-4chbp"
Jul 20 19:34:51.362946 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.362841    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv526\" (UniqueName: \"kubernetes.io/projected/e19a7680-c1b3-49e7-8e02-e3d48ecba89d-kube-api-access-sv526\") pod \"payload-processing-5bf47cd988-4chbp\" (UID: \"e19a7680-c1b3-49e7-8e02-e3d48ecba89d\") " pod="openshift-ingress/payload-processing-5bf47cd988-4chbp"
Jul 20 19:34:51.463625 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.463536    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19a7680-c1b3-49e7-8e02-e3d48ecba89d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-4chbp\" (UID: \"e19a7680-c1b3-49e7-8e02-e3d48ecba89d\") " pod="openshift-ingress/payload-processing-5bf47cd988-4chbp"
Jul 20 19:34:51.463625 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.463609    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv526\" (UniqueName: \"kubernetes.io/projected/e19a7680-c1b3-49e7-8e02-e3d48ecba89d-kube-api-access-sv526\") pod \"payload-processing-5bf47cd988-4chbp\" (UID: \"e19a7680-c1b3-49e7-8e02-e3d48ecba89d\") " pod="openshift-ingress/payload-processing-5bf47cd988-4chbp"
Jul 20 19:34:51.464518 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.464493    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e19a7680-c1b3-49e7-8e02-e3d48ecba89d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-4chbp\" (UID: \"e19a7680-c1b3-49e7-8e02-e3d48ecba89d\") " pod="openshift-ingress/payload-processing-5bf47cd988-4chbp"
Jul 20 19:34:51.471471 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.471446    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv526\" (UniqueName: \"kubernetes.io/projected/e19a7680-c1b3-49e7-8e02-e3d48ecba89d-kube-api-access-sv526\") pod \"payload-processing-5bf47cd988-4chbp\" (UID: \"e19a7680-c1b3-49e7-8e02-e3d48ecba89d\") " pod="openshift-ingress/payload-processing-5bf47cd988-4chbp"
Jul 20 19:34:51.627670 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.627632    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-4chbp"
Jul 20 19:34:51.762569 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:51.762481    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-4chbp"]
Jul 20 19:34:51.765527 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:34:51.765503    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19a7680_c1b3_49e7_8e02_e3d48ecba89d.slice/crio-43283e8bb1238d3ae2828b76eca5cffc14d55172de43fd15f22c75bc9af7eb1d WatchSource:0}: Error finding container 43283e8bb1238d3ae2828b76eca5cffc14d55172de43fd15f22c75bc9af7eb1d: Status 404 returned error can't find the container with id 43283e8bb1238d3ae2828b76eca5cffc14d55172de43fd15f22c75bc9af7eb1d
Jul 20 19:34:52.283134 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:52.283091    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-4chbp" event={"ID":"e19a7680-c1b3-49e7-8e02-e3d48ecba89d","Type":"ContainerStarted","Data":"43283e8bb1238d3ae2828b76eca5cffc14d55172de43fd15f22c75bc9af7eb1d"}
Jul 20 19:34:52.285128 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:52.285083    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt" event={"ID":"524b1c09-1584-48ee-b2aa-bbd0bd65102c","Type":"ContainerStarted","Data":"b1dede92c230584750f97c1a87f139c993307b5b10605ae2dbdd92c5603f055b"}
Jul 20 19:34:54.294813 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:54.294776    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-4chbp" event={"ID":"e19a7680-c1b3-49e7-8e02-e3d48ecba89d","Type":"ContainerStarted","Data":"810ba0d7008e94eb89feb2cc06969bf7dfee2e234ff9ff6aa7a2261f21859168"}
Jul 20 19:34:54.295505 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:54.294893    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-4chbp"
Jul 20 19:34:54.296400 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:54.296379    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt" event={"ID":"524b1c09-1584-48ee-b2aa-bbd0bd65102c","Type":"ContainerStarted","Data":"e368cd26e58ea57bc023e444e7789aa1437c4a7d9e74d62450e6c2fa3c11302b"}
Jul 20 19:34:54.296493 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:54.296421    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:34:54.310709 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:34:54.310663    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-4chbp" podStartSLOduration=1.659408103 podStartE2EDuration="3.310647647s" totalImagesPullingTime="1.651239544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:34:54.309559216 +0000 UTC m=+746.395510214" watchObservedRunningTime="2026-07-20 19:34:54.310647647 +0000 UTC m=+746.396598645"
Jul 20 19:35:00.305157 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:00.305130    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt"
Jul 20 19:35:00.320528 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:00.320474    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-9f859c8f8-vf9lt" podStartSLOduration=8.182588489 podStartE2EDuration="10.32045874s" totalImagesPullingTime="2.137870251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:34:54.3246756 +0000 UTC m=+746.410626609" watchObservedRunningTime="2026-07-20 19:35:00.32045874 +0000 UTC m=+752.406409737"
Jul 20 19:35:05.302422 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:05.302393    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-4chbp"
Jul 20 19:35:24.109415 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.109376    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"]
Jul 20 19:35:24.112882 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.112860    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.115000 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.114980    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 20 19:35:24.115111 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.114980    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dxpmb\""
Jul 20 19:35:24.115111 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.115014    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 20 19:35:24.115722 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.115705    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 20 19:35:24.122113 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.122094    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"]
Jul 20 19:35:24.246421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.246388    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.246421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.246426    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.246662 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.246448    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5sw7\" (UniqueName: \"kubernetes.io/projected/198f17cb-6252-420a-a634-c2f4419ce9e9-kube-api-access-f5sw7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.246662 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.246566    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198f17cb-6252-420a-a634-c2f4419ce9e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.246662 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.246604    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.246787 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.246660    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.347917 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.347885    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.348084 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.347952    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.348084 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.347979    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.348084 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.348005    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5sw7\" (UniqueName: \"kubernetes.io/projected/198f17cb-6252-420a-a634-c2f4419ce9e9-kube-api-access-f5sw7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.348084 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.348034    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198f17cb-6252-420a-a634-c2f4419ce9e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.348084 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.348060    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.348431 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.348408    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.348497 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.348429    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.348497 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.348438    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.350283 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.350239    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/198f17cb-6252-420a-a634-c2f4419ce9e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.350595 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.350579    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/198f17cb-6252-420a-a634-c2f4419ce9e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.356162 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.356136    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5sw7\" (UniqueName: \"kubernetes.io/projected/198f17cb-6252-420a-a634-c2f4419ce9e9-kube-api-access-f5sw7\") pod \"e2e-trlp-test-simulated-kserve-598466548d-wrxnh\" (UID: \"198f17cb-6252-420a-a634-c2f4419ce9e9\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.423790 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.423757    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:24.553173 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:24.553131    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"]
Jul 20 19:35:24.555467 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:35:24.555430    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198f17cb_6252_420a_a634_c2f4419ce9e9.slice/crio-64c7a5a37183c9931952822635a9c1f0cf849a7ef4938d7664d8d5d6bf26b9d1 WatchSource:0}: Error finding container 64c7a5a37183c9931952822635a9c1f0cf849a7ef4938d7664d8d5d6bf26b9d1: Status 404 returned error can't find the container with id 64c7a5a37183c9931952822635a9c1f0cf849a7ef4938d7664d8d5d6bf26b9d1
Jul 20 19:35:25.423083 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:25.423041    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh" event={"ID":"198f17cb-6252-420a-a634-c2f4419ce9e9","Type":"ContainerStarted","Data":"64c7a5a37183c9931952822635a9c1f0cf849a7ef4938d7664d8d5d6bf26b9d1"}
Jul 20 19:35:26.428437 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:26.428401    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh" event={"ID":"198f17cb-6252-420a-a634-c2f4419ce9e9","Type":"ContainerStarted","Data":"1a2a602008fd1735e57f36c144fa37b190d99db59efdbb8c6b3f0bef30691af7"}
Jul 20 19:35:26.446784 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:26.446734    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh" podStartSLOduration=1.4246081369999999 podStartE2EDuration="2.446718835s" totalImagesPullingTime="1.022110698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:35:26.445871471 +0000 UTC m=+778.531822482" watchObservedRunningTime="2026-07-20 19:35:26.446718835 +0000 UTC m=+778.532669832"
Jul 20 19:35:34.424667 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:34.424627    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:34.425325 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:34.424698    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:34.437202 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:34.437176    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:35:34.474204 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:35:34.474175    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-wrxnh"
Jul 20 19:37:28.377164 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:28.377085    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:37:28.377789 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:28.377767    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:37:33.151487 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.151449    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85756d6b4-c2zbv"]
Jul 20 19:37:33.154729 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.154705    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85756d6b4-c2zbv"
Jul 20 19:37:33.164714 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.164691    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85756d6b4-c2zbv"]
Jul 20 19:37:33.184863 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.184828    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d82fd48a-0bd3-4367-a994-6079e3cf5897-tls-cert\") pod \"authorino-85756d6b4-c2zbv\" (UID: \"d82fd48a-0bd3-4367-a994-6079e3cf5897\") " pod="kuadrant-system/authorino-85756d6b4-c2zbv"
Jul 20 19:37:33.184863 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.184864    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8ndf\" (UniqueName: \"kubernetes.io/projected/d82fd48a-0bd3-4367-a994-6079e3cf5897-kube-api-access-x8ndf\") pod \"authorino-85756d6b4-c2zbv\" (UID: \"d82fd48a-0bd3-4367-a994-6079e3cf5897\") " pod="kuadrant-system/authorino-85756d6b4-c2zbv"
Jul 20 19:37:33.286052 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.286014    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d82fd48a-0bd3-4367-a994-6079e3cf5897-tls-cert\") pod \"authorino-85756d6b4-c2zbv\" (UID: \"d82fd48a-0bd3-4367-a994-6079e3cf5897\") " pod="kuadrant-system/authorino-85756d6b4-c2zbv"
Jul 20 19:37:33.286290 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.286069    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8ndf\" (UniqueName: \"kubernetes.io/projected/d82fd48a-0bd3-4367-a994-6079e3cf5897-kube-api-access-x8ndf\") pod \"authorino-85756d6b4-c2zbv\" (UID: \"d82fd48a-0bd3-4367-a994-6079e3cf5897\") " pod="kuadrant-system/authorino-85756d6b4-c2zbv"
Jul 20 19:37:33.288529 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.288510    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d82fd48a-0bd3-4367-a994-6079e3cf5897-tls-cert\") pod \"authorino-85756d6b4-c2zbv\" (UID: \"d82fd48a-0bd3-4367-a994-6079e3cf5897\") " pod="kuadrant-system/authorino-85756d6b4-c2zbv"
Jul 20 19:37:33.293659 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.293641    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8ndf\" (UniqueName: \"kubernetes.io/projected/d82fd48a-0bd3-4367-a994-6079e3cf5897-kube-api-access-x8ndf\") pod \"authorino-85756d6b4-c2zbv\" (UID: \"d82fd48a-0bd3-4367-a994-6079e3cf5897\") " pod="kuadrant-system/authorino-85756d6b4-c2zbv"
Jul 20 19:37:33.463419 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.463336    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85756d6b4-c2zbv"
Jul 20 19:37:33.588700 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.588662    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85756d6b4-c2zbv"]
Jul 20 19:37:33.591788 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:37:33.591763    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd82fd48a_0bd3_4367_a994_6079e3cf5897.slice/crio-298ab4d796da0e84f6f718127d286b27dea176d3ed8569c89834fca737c2358a WatchSource:0}: Error finding container 298ab4d796da0e84f6f718127d286b27dea176d3ed8569c89834fca737c2358a: Status 404 returned error can't find the container with id 298ab4d796da0e84f6f718127d286b27dea176d3ed8569c89834fca737c2358a
Jul 20 19:37:33.953209 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:33.953175    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85756d6b4-c2zbv" event={"ID":"d82fd48a-0bd3-4367-a994-6079e3cf5897","Type":"ContainerStarted","Data":"298ab4d796da0e84f6f718127d286b27dea176d3ed8569c89834fca737c2358a"}
Jul 20 19:37:34.960298 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:34.960240    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85756d6b4-c2zbv" event={"ID":"d82fd48a-0bd3-4367-a994-6079e3cf5897","Type":"ContainerStarted","Data":"31c9b6e0c9f8103d5d14303241ca9e5f9a8b085d66012d5967907f4ac977553c"}
Jul 20 19:37:34.983590 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:34.983531    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85756d6b4-c2zbv" podStartSLOduration=1.444222803 podStartE2EDuration="1.983514483s" totalImagesPullingTime="539.29168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:37:34.982032455 +0000 UTC m=+907.067983454" watchObservedRunningTime="2026-07-20 19:37:34.983514483 +0000 UTC m=+907.069465541"
Jul 20 19:37:35.009421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.009388    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dd847567c-k9bbj"]
Jul 20 19:37:35.009665 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.009576    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-dd847567c-k9bbj" podUID="496b8a76-ac44-4525-b4d3-8c7c32e409c0" containerName="authorino" containerID="cri-o://f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636" gracePeriod=30
Jul 20 19:37:35.264870 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.264845    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dd847567c-k9bbj"
Jul 20 19:37:35.302499 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.302471    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/496b8a76-ac44-4525-b4d3-8c7c32e409c0-tls-cert\") pod \"496b8a76-ac44-4525-b4d3-8c7c32e409c0\" (UID: \"496b8a76-ac44-4525-b4d3-8c7c32e409c0\") "
Jul 20 19:37:35.302680 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.302541    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjd77\" (UniqueName: \"kubernetes.io/projected/496b8a76-ac44-4525-b4d3-8c7c32e409c0-kube-api-access-tjd77\") pod \"496b8a76-ac44-4525-b4d3-8c7c32e409c0\" (UID: \"496b8a76-ac44-4525-b4d3-8c7c32e409c0\") "
Jul 20 19:37:35.304641 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.304607    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496b8a76-ac44-4525-b4d3-8c7c32e409c0-kube-api-access-tjd77" (OuterVolumeSpecName: "kube-api-access-tjd77") pod "496b8a76-ac44-4525-b4d3-8c7c32e409c0" (UID: "496b8a76-ac44-4525-b4d3-8c7c32e409c0"). InnerVolumeSpecName "kube-api-access-tjd77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:37:35.312333 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.312298    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496b8a76-ac44-4525-b4d3-8c7c32e409c0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "496b8a76-ac44-4525-b4d3-8c7c32e409c0" (UID: "496b8a76-ac44-4525-b4d3-8c7c32e409c0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:37:35.403960 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.403924    2550 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/496b8a76-ac44-4525-b4d3-8c7c32e409c0-tls-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:37:35.403960 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.403954    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjd77\" (UniqueName: \"kubernetes.io/projected/496b8a76-ac44-4525-b4d3-8c7c32e409c0-kube-api-access-tjd77\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:37:35.966666 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.966635    2550 generic.go:356] "Generic (PLEG): container finished" podID="496b8a76-ac44-4525-b4d3-8c7c32e409c0" containerID="f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636" exitCode=0
Jul 20 19:37:35.967126 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.966682    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dd847567c-k9bbj"
Jul 20 19:37:35.967126 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.966725    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dd847567c-k9bbj" event={"ID":"496b8a76-ac44-4525-b4d3-8c7c32e409c0","Type":"ContainerDied","Data":"f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636"}
Jul 20 19:37:35.967126 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.966771    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dd847567c-k9bbj" event={"ID":"496b8a76-ac44-4525-b4d3-8c7c32e409c0","Type":"ContainerDied","Data":"966e511797fc59bf7f3f18ec5e0d06d59c839fe34b40ef9b29c3d3ecfa1e4bd6"}
Jul 20 19:37:35.967126 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.966796    2550 scope.go:117] "RemoveContainer" containerID="f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636"
Jul 20 19:37:35.976735 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.976707    2550 scope.go:117] "RemoveContainer" containerID="f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636"
Jul 20 19:37:35.977020 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:37:35.976999    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636\": container with ID starting with f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636 not found: ID does not exist" containerID="f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636"
Jul 20 19:37:35.977096 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.977028    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636"} err="failed to get container status \"f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636\": rpc error: code = NotFound desc = could not find container \"f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636\": container with ID starting with f93ed20ec1250c6993be1db4cf8b13a280628a361cf824035699d2a44b079636 not found: ID does not exist"
Jul 20 19:37:35.989247 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.989218    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dd847567c-k9bbj"]
Jul 20 19:37:35.995138 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:35.995107    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-dd847567c-k9bbj"]
Jul 20 19:37:36.450158 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:37:36.450114    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="496b8a76-ac44-4525-b4d3-8c7c32e409c0" path="/var/lib/kubelet/pods/496b8a76-ac44-4525-b4d3-8c7c32e409c0/volumes"
Jul 20 19:38:23.498210 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:23.498166    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-cwstv"]
Jul 20 19:38:24.169963 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:24.169919    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75c5bf7b68-cwstv" podUID="641b18fb-85f7-438a-a43b-2318992db0c9" containerName="manager" containerID="cri-o://032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c" gracePeriod=30
Jul 20 19:38:24.416534 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:24.416506    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:38:24.430888 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:24.430815    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljkjc\" (UniqueName: \"kubernetes.io/projected/641b18fb-85f7-438a-a43b-2318992db0c9-kube-api-access-ljkjc\") pod \"641b18fb-85f7-438a-a43b-2318992db0c9\" (UID: \"641b18fb-85f7-438a-a43b-2318992db0c9\") "
Jul 20 19:38:24.431020 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:24.430896    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/641b18fb-85f7-438a-a43b-2318992db0c9-webhook-cert\") pod \"641b18fb-85f7-438a-a43b-2318992db0c9\" (UID: \"641b18fb-85f7-438a-a43b-2318992db0c9\") "
Jul 20 19:38:24.432817 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:24.432791    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641b18fb-85f7-438a-a43b-2318992db0c9-kube-api-access-ljkjc" (OuterVolumeSpecName: "kube-api-access-ljkjc") pod "641b18fb-85f7-438a-a43b-2318992db0c9" (UID: "641b18fb-85f7-438a-a43b-2318992db0c9"). InnerVolumeSpecName "kube-api-access-ljkjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:38:24.432919 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:24.432822    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641b18fb-85f7-438a-a43b-2318992db0c9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "641b18fb-85f7-438a-a43b-2318992db0c9" (UID: "641b18fb-85f7-438a-a43b-2318992db0c9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:38:24.531844 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:24.531809    2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/641b18fb-85f7-438a-a43b-2318992db0c9-webhook-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:38:24.531844 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:24.531837    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljkjc\" (UniqueName: \"kubernetes.io/projected/641b18fb-85f7-438a-a43b-2318992db0c9-kube-api-access-ljkjc\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:38:25.175510 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:25.175479    2550 generic.go:356] "Generic (PLEG): container finished" podID="641b18fb-85f7-438a-a43b-2318992db0c9" containerID="032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c" exitCode=0
Jul 20 19:38:25.175696 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:25.175523    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5bf7b68-cwstv"
Jul 20 19:38:25.175696 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:25.175532    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5bf7b68-cwstv" event={"ID":"641b18fb-85f7-438a-a43b-2318992db0c9","Type":"ContainerDied","Data":"032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c"}
Jul 20 19:38:25.175696 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:25.175574    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5bf7b68-cwstv" event={"ID":"641b18fb-85f7-438a-a43b-2318992db0c9","Type":"ContainerDied","Data":"3418050dae2be8bfe0e2b8e3e980a81fc77a5e29072f66355794e41dfde32011"}
Jul 20 19:38:25.175696 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:25.175595    2550 scope.go:117] "RemoveContainer" containerID="032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c"
Jul 20 19:38:25.183567 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:25.183548    2550 scope.go:117] "RemoveContainer" containerID="032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c"
Jul 20 19:38:25.183815 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:38:25.183797    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c\": container with ID starting with 032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c not found: ID does not exist" containerID="032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c"
Jul 20 19:38:25.183864 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:25.183823    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c"} err="failed to get container status \"032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c\": rpc error: code = NotFound desc = could not find container \"032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c\": container with ID starting with 032a1e1865a54e2b82b35f35529141f5ef3a0756885a52ff37966d3a729e5b6c not found: ID does not exist"
Jul 20 19:38:25.190682 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:25.190658    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-cwstv"]
Jul 20 19:38:25.194599 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:25.194579    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-cwstv"]
Jul 20 19:38:26.449938 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:26.449907    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="641b18fb-85f7-438a-a43b-2318992db0c9" path="/var/lib/kubelet/pods/641b18fb-85f7-438a-a43b-2318992db0c9/volumes"
Jul 20 19:38:55.524673 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.524589    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-774c66cdbb-fvh8v"]
Jul 20 19:38:55.525227 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.525042    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496b8a76-ac44-4525-b4d3-8c7c32e409c0" containerName="authorino"
Jul 20 19:38:55.525227 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.525062    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="496b8a76-ac44-4525-b4d3-8c7c32e409c0" containerName="authorino"
Jul 20 19:38:55.525227 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.525075    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641b18fb-85f7-438a-a43b-2318992db0c9" containerName="manager"
Jul 20 19:38:55.525227 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.525082    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="641b18fb-85f7-438a-a43b-2318992db0c9" containerName="manager"
Jul 20 19:38:55.530342 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.530174    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="641b18fb-85f7-438a-a43b-2318992db0c9" containerName="manager"
Jul 20 19:38:55.530342 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.530226    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="496b8a76-ac44-4525-b4d3-8c7c32e409c0" containerName="authorino"
Jul 20 19:38:55.533547 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.533527    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.541608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.541582    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774c66cdbb-fvh8v"]
Jul 20 19:38:55.689883 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.689835    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-service-ca\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.690055 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.689902    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-oauth-serving-cert\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.690055 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.689937    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaafa640-41e2-49f3-a712-840711843ac9-console-oauth-config\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.690055 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.689959    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-trusted-ca-bundle\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.690055 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.689979    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7gvm\" (UniqueName: \"kubernetes.io/projected/aaafa640-41e2-49f3-a712-840711843ac9-kube-api-access-p7gvm\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.690055 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.690004    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaafa640-41e2-49f3-a712-840711843ac9-console-serving-cert\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.690055 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.690030    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-console-config\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.791368 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.791292    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-service-ca\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.791368 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.791342    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-oauth-serving-cert\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.791533 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.791440    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaafa640-41e2-49f3-a712-840711843ac9-console-oauth-config\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.791533 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.791459    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-trusted-ca-bundle\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.791533 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.791480    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7gvm\" (UniqueName: \"kubernetes.io/projected/aaafa640-41e2-49f3-a712-840711843ac9-kube-api-access-p7gvm\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.791533 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.791498    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaafa640-41e2-49f3-a712-840711843ac9-console-serving-cert\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.791533 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.791514    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-console-config\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.792186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.792156    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-service-ca\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.792332 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.792156    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-oauth-serving-cert\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.792428 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.792405    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-console-config\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.792495 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.792479    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aaafa640-41e2-49f3-a712-840711843ac9-trusted-ca-bundle\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.794018 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.793996    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aaafa640-41e2-49f3-a712-840711843ac9-console-serving-cert\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.794116 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.794019    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aaafa640-41e2-49f3-a712-840711843ac9-console-oauth-config\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.799080 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.799057    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7gvm\" (UniqueName: \"kubernetes.io/projected/aaafa640-41e2-49f3-a712-840711843ac9-kube-api-access-p7gvm\") pod \"console-774c66cdbb-fvh8v\" (UID: \"aaafa640-41e2-49f3-a712-840711843ac9\") " pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.843731 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.843700    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:38:55.971090 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:55.971057    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774c66cdbb-fvh8v"]
Jul 20 19:38:55.973864 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:38:55.973835    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaafa640_41e2_49f3_a712_840711843ac9.slice/crio-fde0d98f2fb8c97a044e42b3c28af5aaf7edd34773def83db5aca208f5f9bc6b WatchSource:0}: Error finding container fde0d98f2fb8c97a044e42b3c28af5aaf7edd34773def83db5aca208f5f9bc6b: Status 404 returned error can't find the container with id fde0d98f2fb8c97a044e42b3c28af5aaf7edd34773def83db5aca208f5f9bc6b
Jul 20 19:38:56.310021 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:56.309975    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774c66cdbb-fvh8v" event={"ID":"aaafa640-41e2-49f3-a712-840711843ac9","Type":"ContainerStarted","Data":"38ae62bc80d6127e4783e4aceb60c986e4b36c914803733198592942dd363d84"}
Jul 20 19:38:56.310021 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:56.310028    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774c66cdbb-fvh8v" event={"ID":"aaafa640-41e2-49f3-a712-840711843ac9","Type":"ContainerStarted","Data":"fde0d98f2fb8c97a044e42b3c28af5aaf7edd34773def83db5aca208f5f9bc6b"}
Jul 20 19:38:56.328165 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:38:56.328115    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-774c66cdbb-fvh8v" podStartSLOduration=1.328098776 podStartE2EDuration="1.328098776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:38:56.326992563 +0000 UTC m=+988.412943561" watchObservedRunningTime="2026-07-20 19:38:56.328098776 +0000 UTC m=+988.414049773"
Jul 20 19:39:05.844753 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:05.844713    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:39:05.845222 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:05.844808    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:39:05.849507 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:05.849479    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:39:06.355494 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:06.355467    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-774c66cdbb-fvh8v"
Jul 20 19:39:06.399465 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:06.399436    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f578c898-sh7xw"]
Jul 20 19:39:26.004992 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.004950    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-zhc9q"]
Jul 20 19:39:26.008164 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.008144    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:26.011204 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.011170    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tk6n8\""
Jul 20 19:39:26.011204 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.011192    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 20 19:39:26.011204 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.011170    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 20 19:39:26.015745 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.015711    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-zhc9q"]
Jul 20 19:39:26.029660 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.029626    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjqz\" (UniqueName: \"kubernetes.io/projected/9b49a770-0d9f-4a65-a747-530506daf840-kube-api-access-jqjqz\") pod \"maas-controller-75c5bf7b68-zhc9q\" (UID: \"9b49a770-0d9f-4a65-a747-530506daf840\") " pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:26.029856 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.029686    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b49a770-0d9f-4a65-a747-530506daf840-webhook-cert\") pod \"maas-controller-75c5bf7b68-zhc9q\" (UID: \"9b49a770-0d9f-4a65-a747-530506daf840\") " pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:26.130411 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.130373    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjqz\" (UniqueName: \"kubernetes.io/projected/9b49a770-0d9f-4a65-a747-530506daf840-kube-api-access-jqjqz\") pod \"maas-controller-75c5bf7b68-zhc9q\" (UID: \"9b49a770-0d9f-4a65-a747-530506daf840\") " pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:26.130572 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.130434    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b49a770-0d9f-4a65-a747-530506daf840-webhook-cert\") pod \"maas-controller-75c5bf7b68-zhc9q\" (UID: \"9b49a770-0d9f-4a65-a747-530506daf840\") " pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:26.132858 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.132840    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b49a770-0d9f-4a65-a747-530506daf840-webhook-cert\") pod \"maas-controller-75c5bf7b68-zhc9q\" (UID: \"9b49a770-0d9f-4a65-a747-530506daf840\") " pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:26.139200 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.139172    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjqz\" (UniqueName: \"kubernetes.io/projected/9b49a770-0d9f-4a65-a747-530506daf840-kube-api-access-jqjqz\") pod \"maas-controller-75c5bf7b68-zhc9q\" (UID: \"9b49a770-0d9f-4a65-a747-530506daf840\") " pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:26.321291 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.321186    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:26.451429 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.451294    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-zhc9q"]
Jul 20 19:39:26.454177 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:39:26.454146    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b49a770_0d9f_4a65_a747_530506daf840.slice/crio-a76aab3c5ab01c2b86b149849722d1652654b0b0f70ec61fe36dc3e16dfadc9a WatchSource:0}: Error finding container a76aab3c5ab01c2b86b149849722d1652654b0b0f70ec61fe36dc3e16dfadc9a: Status 404 returned error can't find the container with id a76aab3c5ab01c2b86b149849722d1652654b0b0f70ec61fe36dc3e16dfadc9a
Jul 20 19:39:26.455862 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:26.455843    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 19:39:27.439726 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:27.439693    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q" event={"ID":"9b49a770-0d9f-4a65-a747-530506daf840","Type":"ContainerStarted","Data":"d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0"}
Jul 20 19:39:27.440145 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:27.439747    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q" event={"ID":"9b49a770-0d9f-4a65-a747-530506daf840","Type":"ContainerStarted","Data":"a76aab3c5ab01c2b86b149849722d1652654b0b0f70ec61fe36dc3e16dfadc9a"}
Jul 20 19:39:27.440145 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:27.439777    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:27.459510 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:27.459461    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q" podStartSLOduration=2.00757482 podStartE2EDuration="2.45944468s" totalImagesPullingTime="451.86986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:39:27.457192693 +0000 UTC m=+1019.543143688" watchObservedRunningTime="2026-07-20 19:39:27.45944468 +0000 UTC m=+1019.545395677"
Jul 20 19:39:31.419298 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.419238    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55f578c898-sh7xw" podUID="8d1cf28d-721a-435c-bab3-ff1301f9eec7" containerName="console" containerID="cri-o://8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861" gracePeriod=15
Jul 20 19:39:31.661302 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.661275    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:39:31.674723 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.674650    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-oauth-serving-cert\") pod \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") "
Jul 20 19:39:31.674723 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.674691    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-oauth-config\") pod \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") "
Jul 20 19:39:31.674861 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.674728    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-service-ca\") pod \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") "
Jul 20 19:39:31.674861 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.674753    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g988s\" (UniqueName: \"kubernetes.io/projected/8d1cf28d-721a-435c-bab3-ff1301f9eec7-kube-api-access-g988s\") pod \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") "
Jul 20 19:39:31.674861 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.674767    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-config\") pod \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") "
Jul 20 19:39:31.674969 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.674953    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-serving-cert\") pod \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") "
Jul 20 19:39:31.675070 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.675051    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-trusted-ca-bundle\") pod \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\" (UID: \"8d1cf28d-721a-435c-bab3-ff1301f9eec7\") "
Jul 20 19:39:31.675145 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.675120    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8d1cf28d-721a-435c-bab3-ff1301f9eec7" (UID: "8d1cf28d-721a-435c-bab3-ff1301f9eec7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:39:31.675145 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.675126    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-service-ca" (OuterVolumeSpecName: "service-ca") pod "8d1cf28d-721a-435c-bab3-ff1301f9eec7" (UID: "8d1cf28d-721a-435c-bab3-ff1301f9eec7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:39:31.675272 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.675164    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-config" (OuterVolumeSpecName: "console-config") pod "8d1cf28d-721a-435c-bab3-ff1301f9eec7" (UID: "8d1cf28d-721a-435c-bab3-ff1301f9eec7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:39:31.675535 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.675513    2550 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:39:31.675535 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.675536    2550 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-config\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:39:31.675695 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.675549    2550 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-oauth-serving-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:39:31.675695 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.675563    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8d1cf28d-721a-435c-bab3-ff1301f9eec7" (UID: "8d1cf28d-721a-435c-bab3-ff1301f9eec7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:39:31.677105 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.677073    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8d1cf28d-721a-435c-bab3-ff1301f9eec7" (UID: "8d1cf28d-721a-435c-bab3-ff1301f9eec7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:39:31.677194 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.677119    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8d1cf28d-721a-435c-bab3-ff1301f9eec7" (UID: "8d1cf28d-721a-435c-bab3-ff1301f9eec7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:39:31.677194 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.677129    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1cf28d-721a-435c-bab3-ff1301f9eec7-kube-api-access-g988s" (OuterVolumeSpecName: "kube-api-access-g988s") pod "8d1cf28d-721a-435c-bab3-ff1301f9eec7" (UID: "8d1cf28d-721a-435c-bab3-ff1301f9eec7"). InnerVolumeSpecName "kube-api-access-g988s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:39:31.776628 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.776589    2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d1cf28d-721a-435c-bab3-ff1301f9eec7-trusted-ca-bundle\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:39:31.776628 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.776623    2550 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-oauth-config\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:39:31.776628 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.776633    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g988s\" (UniqueName: \"kubernetes.io/projected/8d1cf28d-721a-435c-bab3-ff1301f9eec7-kube-api-access-g988s\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:39:31.776876 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:31.776643    2550 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d1cf28d-721a-435c-bab3-ff1301f9eec7-console-serving-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:39:32.461062 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:32.461032    2550 generic.go:356] "Generic (PLEG): container finished" podID="8d1cf28d-721a-435c-bab3-ff1301f9eec7" containerID="8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861" exitCode=0
Jul 20 19:39:32.461424 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:32.461076    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f578c898-sh7xw"
Jul 20 19:39:32.461424 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:32.461118    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f578c898-sh7xw" event={"ID":"8d1cf28d-721a-435c-bab3-ff1301f9eec7","Type":"ContainerDied","Data":"8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861"}
Jul 20 19:39:32.461424 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:32.461170    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f578c898-sh7xw" event={"ID":"8d1cf28d-721a-435c-bab3-ff1301f9eec7","Type":"ContainerDied","Data":"bbf9f2beada72d8df4b62ef518010921ac61ad796e1231cb517044172bc2c188"}
Jul 20 19:39:32.461424 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:32.461197    2550 scope.go:117] "RemoveContainer" containerID="8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861"
Jul 20 19:39:32.469499 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:32.469479    2550 scope.go:117] "RemoveContainer" containerID="8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861"
Jul 20 19:39:32.469781 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:39:32.469761    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861\": container with ID starting with 8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861 not found: ID does not exist" containerID="8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861"
Jul 20 19:39:32.469829 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:32.469792    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861"} err="failed to get container status \"8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861\": rpc error: code = NotFound desc = could not find container \"8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861\": container with ID starting with 8effe58932e61314df41037587b17ee46102a5ac31a576c7803b87de5d754861 not found: ID does not exist"
Jul 20 19:39:32.476475 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:32.476447    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f578c898-sh7xw"]
Jul 20 19:39:32.479677 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:32.479653    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55f578c898-sh7xw"]
Jul 20 19:39:34.450549 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:34.450504    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d1cf28d-721a-435c-bab3-ff1301f9eec7" path="/var/lib/kubelet/pods/8d1cf28d-721a-435c-bab3-ff1301f9eec7/volumes"
Jul 20 19:39:38.451809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:38.451775    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:43.102832 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.102779    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-zhc9q"]
Jul 20 19:39:43.103296 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.103036    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q" podUID="9b49a770-0d9f-4a65-a747-530506daf840" containerName="manager" containerID="cri-o://d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0" gracePeriod=30
Jul 20 19:39:43.357119 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.357059    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:43.474388 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.474349    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b49a770-0d9f-4a65-a747-530506daf840-webhook-cert\") pod \"9b49a770-0d9f-4a65-a747-530506daf840\" (UID: \"9b49a770-0d9f-4a65-a747-530506daf840\") "
Jul 20 19:39:43.474600 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.474406    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqjqz\" (UniqueName: \"kubernetes.io/projected/9b49a770-0d9f-4a65-a747-530506daf840-kube-api-access-jqjqz\") pod \"9b49a770-0d9f-4a65-a747-530506daf840\" (UID: \"9b49a770-0d9f-4a65-a747-530506daf840\") "
Jul 20 19:39:43.476511 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.476471    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b49a770-0d9f-4a65-a747-530506daf840-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9b49a770-0d9f-4a65-a747-530506daf840" (UID: "9b49a770-0d9f-4a65-a747-530506daf840"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:39:43.476634 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.476589    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b49a770-0d9f-4a65-a747-530506daf840-kube-api-access-jqjqz" (OuterVolumeSpecName: "kube-api-access-jqjqz") pod "9b49a770-0d9f-4a65-a747-530506daf840" (UID: "9b49a770-0d9f-4a65-a747-530506daf840"). InnerVolumeSpecName "kube-api-access-jqjqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:39:43.510135 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.510100    2550 generic.go:356] "Generic (PLEG): container finished" podID="9b49a770-0d9f-4a65-a747-530506daf840" containerID="d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0" exitCode=0
Jul 20 19:39:43.510362 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.510157    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q"
Jul 20 19:39:43.510362 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.510196    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q" event={"ID":"9b49a770-0d9f-4a65-a747-530506daf840","Type":"ContainerDied","Data":"d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0"}
Jul 20 19:39:43.510362 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.510278    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5bf7b68-zhc9q" event={"ID":"9b49a770-0d9f-4a65-a747-530506daf840","Type":"ContainerDied","Data":"a76aab3c5ab01c2b86b149849722d1652654b0b0f70ec61fe36dc3e16dfadc9a"}
Jul 20 19:39:43.510362 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.510290    2550 scope.go:117] "RemoveContainer" containerID="d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0"
Jul 20 19:39:43.519415 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.519398    2550 scope.go:117] "RemoveContainer" containerID="d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0"
Jul 20 19:39:43.519669 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:39:43.519649    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0\": container with ID starting with d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0 not found: ID does not exist" containerID="d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0"
Jul 20 19:39:43.519726 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.519678    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0"} err="failed to get container status \"d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0\": rpc error: code = NotFound desc = could not find container \"d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0\": container with ID starting with d4e67b241820e056c9249084ee7f2db525796f4993f1a7a97249390b8b9953b0 not found: ID does not exist"
Jul 20 19:39:43.530894 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.530867    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-zhc9q"]
Jul 20 19:39:43.534614 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.534589    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-zhc9q"]
Jul 20 19:39:43.575441 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.575414    2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b49a770-0d9f-4a65-a747-530506daf840-webhook-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:39:43.575441 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:43.575438    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqjqz\" (UniqueName: \"kubernetes.io/projected/9b49a770-0d9f-4a65-a747-530506daf840-kube-api-access-jqjqz\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:39:44.323845 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.323806    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-9db78"]
Jul 20 19:39:44.324240 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.324115    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b49a770-0d9f-4a65-a747-530506daf840" containerName="manager"
Jul 20 19:39:44.324240 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.324126    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b49a770-0d9f-4a65-a747-530506daf840" containerName="manager"
Jul 20 19:39:44.324240 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.324136    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1cf28d-721a-435c-bab3-ff1301f9eec7" containerName="console"
Jul 20 19:39:44.324240 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.324142    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1cf28d-721a-435c-bab3-ff1301f9eec7" containerName="console"
Jul 20 19:39:44.324240 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.324192    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1cf28d-721a-435c-bab3-ff1301f9eec7" containerName="console"
Jul 20 19:39:44.324240 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.324200    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b49a770-0d9f-4a65-a747-530506daf840" containerName="manager"
Jul 20 19:39:44.328370 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.328352    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5bf7b68-9db78"
Jul 20 19:39:44.331413 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.331388    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 20 19:39:44.331580 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.331390    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 20 19:39:44.331580 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.331391    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tk6n8\""
Jul 20 19:39:44.337177 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.337152    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-9db78"]
Jul 20 19:39:44.383384 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.383346    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkrt2\" (UniqueName: \"kubernetes.io/projected/6de81c58-514d-4d01-9d9b-77e54012c519-kube-api-access-pkrt2\") pod \"maas-controller-75c5bf7b68-9db78\" (UID: \"6de81c58-514d-4d01-9d9b-77e54012c519\") " pod="opendatahub/maas-controller-75c5bf7b68-9db78"
Jul 20 19:39:44.383538 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.383404    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6de81c58-514d-4d01-9d9b-77e54012c519-webhook-cert\") pod \"maas-controller-75c5bf7b68-9db78\" (UID: \"6de81c58-514d-4d01-9d9b-77e54012c519\") " pod="opendatahub/maas-controller-75c5bf7b68-9db78"
Jul 20 19:39:44.450511 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.450478    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b49a770-0d9f-4a65-a747-530506daf840" path="/var/lib/kubelet/pods/9b49a770-0d9f-4a65-a747-530506daf840/volumes"
Jul 20 19:39:44.484413 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.484370    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkrt2\" (UniqueName: \"kubernetes.io/projected/6de81c58-514d-4d01-9d9b-77e54012c519-kube-api-access-pkrt2\") pod \"maas-controller-75c5bf7b68-9db78\" (UID: \"6de81c58-514d-4d01-9d9b-77e54012c519\") " pod="opendatahub/maas-controller-75c5bf7b68-9db78"
Jul 20 19:39:44.484413 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.484420    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6de81c58-514d-4d01-9d9b-77e54012c519-webhook-cert\") pod \"maas-controller-75c5bf7b68-9db78\" (UID: \"6de81c58-514d-4d01-9d9b-77e54012c519\") " pod="opendatahub/maas-controller-75c5bf7b68-9db78"
Jul 20 19:39:44.486947 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.486920    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6de81c58-514d-4d01-9d9b-77e54012c519-webhook-cert\") pod \"maas-controller-75c5bf7b68-9db78\" (UID: \"6de81c58-514d-4d01-9d9b-77e54012c519\") " pod="opendatahub/maas-controller-75c5bf7b68-9db78"
Jul 20 19:39:44.493371 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.493348    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkrt2\" (UniqueName: \"kubernetes.io/projected/6de81c58-514d-4d01-9d9b-77e54012c519-kube-api-access-pkrt2\") pod \"maas-controller-75c5bf7b68-9db78\" (UID: \"6de81c58-514d-4d01-9d9b-77e54012c519\") " pod="opendatahub/maas-controller-75c5bf7b68-9db78"
Jul 20 19:39:44.640521 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.640491    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5bf7b68-9db78"
Jul 20 19:39:44.770402 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:44.770345    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c5bf7b68-9db78"]
Jul 20 19:39:44.772659 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:39:44.772629    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de81c58_514d_4d01_9d9b_77e54012c519.slice/crio-401773f9d2b91bf5da170a34e477b1710ae5f4e53d64a7d7b38a219482417cdc WatchSource:0}: Error finding container 401773f9d2b91bf5da170a34e477b1710ae5f4e53d64a7d7b38a219482417cdc: Status 404 returned error can't find the container with id 401773f9d2b91bf5da170a34e477b1710ae5f4e53d64a7d7b38a219482417cdc
Jul 20 19:39:45.521243 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:45.521207    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5bf7b68-9db78" event={"ID":"6de81c58-514d-4d01-9d9b-77e54012c519","Type":"ContainerStarted","Data":"cdc1f278b40f1ab2dd5b3d7ce7e97eee19306d286254df621610afbbbef70757"}
Jul 20 19:39:45.521637 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:45.521290    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5bf7b68-9db78" event={"ID":"6de81c58-514d-4d01-9d9b-77e54012c519","Type":"ContainerStarted","Data":"401773f9d2b91bf5da170a34e477b1710ae5f4e53d64a7d7b38a219482417cdc"}
Jul 20 19:39:45.521637 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:45.521313    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75c5bf7b68-9db78"
Jul 20 19:39:45.538619 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:45.538560    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75c5bf7b68-9db78" podStartSLOduration=0.981560291 podStartE2EDuration="1.538541274s" totalImagesPullingTime="556.980983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:39:45.537059725 +0000 UTC m=+1037.623010724" watchObservedRunningTime="2026-07-20 19:39:45.538541274 +0000 UTC m=+1037.624492271"
Jul 20 19:39:56.529229 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:39:56.529193    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75c5bf7b68-9db78"
Jul 20 19:42:28.400561 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:42:28.400528    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:42:28.403009 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:42:28.402984    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:47:28.423351 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:47:28.423320    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:47:28.426274 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:47:28.426235    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:48:15.823547 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:15.823459    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"]
Jul 20 19:48:15.826335 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:15.826313    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:15.828475 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:15.828456    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bf5232f1-kserve-self-signed-certs\""
Jul 20 19:48:15.847968 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:15.847944    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"]
Jul 20 19:48:15.922021 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:15.921987    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce93231-0767-45f3-926a-fa7a68450517-tls-certs\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:15.922180 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:15.922025    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8sqm\" (UniqueName: \"kubernetes.io/projected/fce93231-0767-45f3-926a-fa7a68450517-kube-api-access-s8sqm\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:15.922180 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:15.922100    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-tmp-dir\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:15.922180 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:15.922146    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-model-cache\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:15.922344 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:15.922202    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-home\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:15.922344 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:15.922232    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-dshm\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.022836 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.022801    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-tmp-dir\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.022836 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.022838    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-model-cache\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.023130 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.022867    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-home\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.023130 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.022968    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-dshm\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.023130 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.023014    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce93231-0767-45f3-926a-fa7a68450517-tls-certs\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.023130 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.023035    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8sqm\" (UniqueName: \"kubernetes.io/projected/fce93231-0767-45f3-926a-fa7a68450517-kube-api-access-s8sqm\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.023365 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.023222    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-model-cache\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.023365 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.023332    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-tmp-dir\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.023365 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.023346    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-home\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.025307 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.025286    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-dshm\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.025478 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.025460    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce93231-0767-45f3-926a-fa7a68450517-tls-certs\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.032709 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.032688    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8sqm\" (UniqueName: \"kubernetes.io/projected/fce93231-0767-45f3-926a-fa7a68450517-kube-api-access-s8sqm\") pod \"e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") " pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.135550 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.135502    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:16.263039 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.263004    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"]
Jul 20 19:48:16.266105 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:48:16.266078    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfce93231_0767_45f3_926a_fa7a68450517.slice/crio-906ceb83133ef9711c74622df4ef8fbab6cff21d9ca410c4b4c90de2938f836a WatchSource:0}: Error finding container 906ceb83133ef9711c74622df4ef8fbab6cff21d9ca410c4b4c90de2938f836a: Status 404 returned error can't find the container with id 906ceb83133ef9711c74622df4ef8fbab6cff21d9ca410c4b4c90de2938f836a
Jul 20 19:48:16.671176 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.671145    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc" event={"ID":"fce93231-0767-45f3-926a-fa7a68450517","Type":"ContainerStarted","Data":"2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a"}
Jul 20 19:48:16.671362 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.671184    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc" event={"ID":"fce93231-0767-45f3-926a-fa7a68450517","Type":"ContainerStarted","Data":"906ceb83133ef9711c74622df4ef8fbab6cff21d9ca410c4b4c90de2938f836a"}
Jul 20 19:48:16.689211 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:16.689161    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc" podStartSLOduration=1.689145898 podStartE2EDuration="1.689145898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:48:16.687514571 +0000 UTC m=+1548.773465568" watchObservedRunningTime="2026-07-20 19:48:16.689145898 +0000 UTC m=+1548.775096946"
Jul 20 19:48:26.135719 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:26.135670    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:26.135719 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:26.135726    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:26.147834 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:26.147815    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:26.724148 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:26.724115    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:48:29.218845 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.218810    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"]
Jul 20 19:48:29.221956 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.221936    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.224356 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.224331    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-bf5232f1-kserve-self-signed-certs\""
Jul 20 19:48:29.232696 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.232672    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"]
Jul 20 19:48:29.330130 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.330098    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlcpb\" (UniqueName: \"kubernetes.io/projected/a71bdb13-57de-471a-a7d9-0c28f40bb973-kube-api-access-wlcpb\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.330303 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.330148    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-tmp-dir\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.330303 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.330170    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-home\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.330303 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.330190    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71bdb13-57de-471a-a7d9-0c28f40bb973-tls-certs\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.330303 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.330225    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-dshm\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.330303 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.330242    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-model-cache\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.431029 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.430992    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-dshm\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.431029 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.431030    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-model-cache\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.431287 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.431166    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlcpb\" (UniqueName: \"kubernetes.io/projected/a71bdb13-57de-471a-a7d9-0c28f40bb973-kube-api-access-wlcpb\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.431287 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.431251    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-tmp-dir\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.431409 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.431323    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-home\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.431409 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.431348    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71bdb13-57de-471a-a7d9-0c28f40bb973-tls-certs\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.431409 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.431396    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-model-cache\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.431549 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.431507    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-tmp-dir\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.431629 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.431609    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-home\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.433157 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.433137    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-dshm\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.433768 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.433749    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71bdb13-57de-471a-a7d9-0c28f40bb973-tls-certs\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.438574 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.438554    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlcpb\" (UniqueName: \"kubernetes.io/projected/a71bdb13-57de-471a-a7d9-0c28f40bb973-kube-api-access-wlcpb\") pod \"e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") " pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.531300 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.531218    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:29.659829 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.659794    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"]
Jul 20 19:48:29.662542 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:48:29.662511    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda71bdb13_57de_471a_a7d9_0c28f40bb973.slice/crio-4599dfa4e91b1d7edac13364fed8ba1abf5919f157175a04ab8329f61f9028e4 WatchSource:0}: Error finding container 4599dfa4e91b1d7edac13364fed8ba1abf5919f157175a04ab8329f61f9028e4: Status 404 returned error can't find the container with id 4599dfa4e91b1d7edac13364fed8ba1abf5919f157175a04ab8329f61f9028e4
Jul 20 19:48:29.727980 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.727948    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz" event={"ID":"a71bdb13-57de-471a-a7d9-0c28f40bb973","Type":"ContainerStarted","Data":"b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c"}
Jul 20 19:48:29.728160 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.727990    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz" event={"ID":"a71bdb13-57de-471a-a7d9-0c28f40bb973","Type":"ContainerStarted","Data":"4599dfa4e91b1d7edac13364fed8ba1abf5919f157175a04ab8329f61f9028e4"}
Jul 20 19:48:29.747062 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:29.747013    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz" podStartSLOduration=0.74699586 podStartE2EDuration="746.99586ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:48:29.746153278 +0000 UTC m=+1561.832104279" watchObservedRunningTime="2026-07-20 19:48:29.74699586 +0000 UTC m=+1561.832946857"
Jul 20 19:48:39.531962 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:39.531920    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:39.532399 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:39.531995    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:39.544338 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:39.544313    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:39.784300 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:39.784203    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:48:54.477086 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:54.476273    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"]
Jul 20 19:48:55.999201 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:48:55.999162    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"]
Jul 20 19:49:09.499683 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.499620    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc" podUID="fce93231-0767-45f3-926a-fa7a68450517" containerName="main" containerID="cri-o://2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a" gracePeriod=45
Jul 20 19:49:09.738508 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.738483    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:49:09.787495 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.787417    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-dshm\") pod \"fce93231-0767-45f3-926a-fa7a68450517\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") "
Jul 20 19:49:09.787495 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.787462    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8sqm\" (UniqueName: \"kubernetes.io/projected/fce93231-0767-45f3-926a-fa7a68450517-kube-api-access-s8sqm\") pod \"fce93231-0767-45f3-926a-fa7a68450517\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") "
Jul 20 19:49:09.787495 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.787479    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-tmp-dir\") pod \"fce93231-0767-45f3-926a-fa7a68450517\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") "
Jul 20 19:49:09.787760 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.787501    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-model-cache\") pod \"fce93231-0767-45f3-926a-fa7a68450517\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") "
Jul 20 19:49:09.787760 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.787529    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-home\") pod \"fce93231-0767-45f3-926a-fa7a68450517\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") "
Jul 20 19:49:09.787760 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.787553    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce93231-0767-45f3-926a-fa7a68450517-tls-certs\") pod \"fce93231-0767-45f3-926a-fa7a68450517\" (UID: \"fce93231-0767-45f3-926a-fa7a68450517\") "
Jul 20 19:49:09.787885 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.787776    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-model-cache" (OuterVolumeSpecName: "model-cache") pod "fce93231-0767-45f3-926a-fa7a68450517" (UID: "fce93231-0767-45f3-926a-fa7a68450517"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:49:09.787885 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.787791    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fce93231-0767-45f3-926a-fa7a68450517" (UID: "fce93231-0767-45f3-926a-fa7a68450517"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:49:09.787885 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.787809    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-home" (OuterVolumeSpecName: "home") pod "fce93231-0767-45f3-926a-fa7a68450517" (UID: "fce93231-0767-45f3-926a-fa7a68450517"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:49:09.789653 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.789620    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce93231-0767-45f3-926a-fa7a68450517-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fce93231-0767-45f3-926a-fa7a68450517" (UID: "fce93231-0767-45f3-926a-fa7a68450517"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:49:09.789653 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.789634    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-dshm" (OuterVolumeSpecName: "dshm") pod "fce93231-0767-45f3-926a-fa7a68450517" (UID: "fce93231-0767-45f3-926a-fa7a68450517"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:49:09.789653 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.789648    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce93231-0767-45f3-926a-fa7a68450517-kube-api-access-s8sqm" (OuterVolumeSpecName: "kube-api-access-s8sqm") pod "fce93231-0767-45f3-926a-fa7a68450517" (UID: "fce93231-0767-45f3-926a-fa7a68450517"). InnerVolumeSpecName "kube-api-access-s8sqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:49:09.889160 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.889124    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-dshm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:09.889160 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.889153    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8sqm\" (UniqueName: \"kubernetes.io/projected/fce93231-0767-45f3-926a-fa7a68450517-kube-api-access-s8sqm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:09.889160 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.889164    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-tmp-dir\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:09.889412 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.889173    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-model-cache\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:09.889412 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.889182    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce93231-0767-45f3-926a-fa7a68450517-home\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:09.889412 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.889189    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce93231-0767-45f3-926a-fa7a68450517-tls-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:09.893552 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.893521    2550 generic.go:356] "Generic (PLEG): container finished" podID="fce93231-0767-45f3-926a-fa7a68450517" containerID="2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a" exitCode=0
Jul 20 19:49:09.893671 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.893561    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"
Jul 20 19:49:09.893671 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.893601    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc" event={"ID":"fce93231-0767-45f3-926a-fa7a68450517","Type":"ContainerDied","Data":"2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a"}
Jul 20 19:49:09.893671 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.893636    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc" event={"ID":"fce93231-0767-45f3-926a-fa7a68450517","Type":"ContainerDied","Data":"906ceb83133ef9711c74622df4ef8fbab6cff21d9ca410c4b4c90de2938f836a"}
Jul 20 19:49:09.893671 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.893654    2550 scope.go:117] "RemoveContainer" containerID="2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a"
Jul 20 19:49:09.902279 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.902246    2550 scope.go:117] "RemoveContainer" containerID="2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a"
Jul 20 19:49:09.902556 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:49:09.902539    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a\": container with ID starting with 2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a not found: ID does not exist" containerID="2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a"
Jul 20 19:49:09.902626 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.902563    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a"} err="failed to get container status \"2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a\": rpc error: code = NotFound desc = could not find container \"2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a\": container with ID starting with 2b91e2a23764c9a95583500ebc9a4a285510de1679a80450a6af68f5201a1a9a not found: ID does not exist"
Jul 20 19:49:09.911725 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.911704    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"]
Jul 20 19:49:09.913867 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:09.913845    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bf5232f1-kserve-57cc97bf8c-mzmnc"]
Jul 20 19:49:10.449998 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:10.449965    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce93231-0767-45f3-926a-fa7a68450517" path="/var/lib/kubelet/pods/fce93231-0767-45f3-926a-fa7a68450517/volumes"
Jul 20 19:49:11.018851 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.018816    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz" podUID="a71bdb13-57de-471a-a7d9-0c28f40bb973" containerName="main" containerID="cri-o://b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c" gracePeriod=45
Jul 20 19:49:11.254735 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.254712    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:49:11.300785 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.300719    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlcpb\" (UniqueName: \"kubernetes.io/projected/a71bdb13-57de-471a-a7d9-0c28f40bb973-kube-api-access-wlcpb\") pod \"a71bdb13-57de-471a-a7d9-0c28f40bb973\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") "
Jul 20 19:49:11.300785 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.300769    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71bdb13-57de-471a-a7d9-0c28f40bb973-tls-certs\") pod \"a71bdb13-57de-471a-a7d9-0c28f40bb973\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") "
Jul 20 19:49:11.300968 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.300791    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-model-cache\") pod \"a71bdb13-57de-471a-a7d9-0c28f40bb973\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") "
Jul 20 19:49:11.301022 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.300974    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-tmp-dir\") pod \"a71bdb13-57de-471a-a7d9-0c28f40bb973\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") "
Jul 20 19:49:11.301073 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.301008    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-model-cache" (OuterVolumeSpecName: "model-cache") pod "a71bdb13-57de-471a-a7d9-0c28f40bb973" (UID: "a71bdb13-57de-471a-a7d9-0c28f40bb973"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:49:11.301073 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.301041    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-dshm\") pod \"a71bdb13-57de-471a-a7d9-0c28f40bb973\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") "
Jul 20 19:49:11.301168 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.301114    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-home\") pod \"a71bdb13-57de-471a-a7d9-0c28f40bb973\" (UID: \"a71bdb13-57de-471a-a7d9-0c28f40bb973\") "
Jul 20 19:49:11.301271 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.301223    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a71bdb13-57de-471a-a7d9-0c28f40bb973" (UID: "a71bdb13-57de-471a-a7d9-0c28f40bb973"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:49:11.301397 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.301365    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-home" (OuterVolumeSpecName: "home") pod "a71bdb13-57de-471a-a7d9-0c28f40bb973" (UID: "a71bdb13-57de-471a-a7d9-0c28f40bb973"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:49:11.301449 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.301420    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-model-cache\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:11.301449 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.301439    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-tmp-dir\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:11.302857 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.302839    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a71bdb13-57de-471a-a7d9-0c28f40bb973-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a71bdb13-57de-471a-a7d9-0c28f40bb973" (UID: "a71bdb13-57de-471a-a7d9-0c28f40bb973"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:49:11.302937 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.302880    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71bdb13-57de-471a-a7d9-0c28f40bb973-kube-api-access-wlcpb" (OuterVolumeSpecName: "kube-api-access-wlcpb") pod "a71bdb13-57de-471a-a7d9-0c28f40bb973" (UID: "a71bdb13-57de-471a-a7d9-0c28f40bb973"). InnerVolumeSpecName "kube-api-access-wlcpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:49:11.302937 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.302918    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-dshm" (OuterVolumeSpecName: "dshm") pod "a71bdb13-57de-471a-a7d9-0c28f40bb973" (UID: "a71bdb13-57de-471a-a7d9-0c28f40bb973"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:49:11.402373 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.402337    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-home\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:11.402373 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.402368    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlcpb\" (UniqueName: \"kubernetes.io/projected/a71bdb13-57de-471a-a7d9-0c28f40bb973-kube-api-access-wlcpb\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:11.402373 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.402383    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71bdb13-57de-471a-a7d9-0c28f40bb973-tls-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:11.402584 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.402394    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71bdb13-57de-471a-a7d9-0c28f40bb973-dshm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:49:11.904690 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.904655    2550 generic.go:356] "Generic (PLEG): container finished" podID="a71bdb13-57de-471a-a7d9-0c28f40bb973" containerID="b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c" exitCode=0
Jul 20 19:49:11.904847 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.904711    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"
Jul 20 19:49:11.904847 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.904720    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz" event={"ID":"a71bdb13-57de-471a-a7d9-0c28f40bb973","Type":"ContainerDied","Data":"b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c"}
Jul 20 19:49:11.904847 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.904749    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz" event={"ID":"a71bdb13-57de-471a-a7d9-0c28f40bb973","Type":"ContainerDied","Data":"4599dfa4e91b1d7edac13364fed8ba1abf5919f157175a04ab8329f61f9028e4"}
Jul 20 19:49:11.904847 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.904765    2550 scope.go:117] "RemoveContainer" containerID="b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c"
Jul 20 19:49:11.913756 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.913736    2550 scope.go:117] "RemoveContainer" containerID="b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c"
Jul 20 19:49:11.913992 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:49:11.913971    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c\": container with ID starting with b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c not found: ID does not exist" containerID="b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c"
Jul 20 19:49:11.914055 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.914002    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c"} err="failed to get container status \"b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c\": rpc error: code = NotFound desc = could not find container \"b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c\": container with ID starting with b514dc9bb032179204b0f6aeb3d111b1832df5792ce6d0a1ad47bce49a07e25c not found: ID does not exist"
Jul 20 19:49:11.925020 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.924995    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"]
Jul 20 19:49:11.929154 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:11.929132    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-bf5232f1-kserve-589c44648c-q8fhz"]
Jul 20 19:49:12.450320 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:49:12.450292    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a71bdb13-57de-471a-a7d9-0c28f40bb973" path="/var/lib/kubelet/pods/a71bdb13-57de-471a-a7d9-0c28f40bb973/volumes"
Jul 20 19:52:28.445806 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:28.445774    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:52:28.449625 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:28.449607    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:52:38.766736 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.766704    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"]
Jul 20 19:52:38.767122 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.766986    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce93231-0767-45f3-926a-fa7a68450517" containerName="main"
Jul 20 19:52:38.767122 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.766996    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce93231-0767-45f3-926a-fa7a68450517" containerName="main"
Jul 20 19:52:38.767122 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.767018    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71bdb13-57de-471a-a7d9-0c28f40bb973" containerName="main"
Jul 20 19:52:38.767122 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.767023    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71bdb13-57de-471a-a7d9-0c28f40bb973" containerName="main"
Jul 20 19:52:38.767122 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.767072    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce93231-0767-45f3-926a-fa7a68450517" containerName="main"
Jul 20 19:52:38.767122 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.767078    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71bdb13-57de-471a-a7d9-0c28f40bb973" containerName="main"
Jul 20 19:52:38.769748 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.769728    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:38.771775 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.771757    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4d9731fc-serving-cert\""
Jul 20 19:52:38.782012 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.781992    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"]
Jul 20 19:52:38.899931 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.899888    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46da344-7402-4b5b-a6ce-197c381f10a7-openshift-service-ca\") pod \"maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:38.899931 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.899933    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4lk7\" (UniqueName: \"kubernetes.io/projected/f46da344-7402-4b5b-a6ce-197c381f10a7-kube-api-access-v4lk7\") pod \"maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:38.900153 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:38.899995    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46da344-7402-4b5b-a6ce-197c381f10a7-maas-api-tls\") pod \"maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:39.000806 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:39.000771    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46da344-7402-4b5b-a6ce-197c381f10a7-openshift-service-ca\") pod \"maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:39.000806 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:39.000807    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lk7\" (UniqueName: \"kubernetes.io/projected/f46da344-7402-4b5b-a6ce-197c381f10a7-kube-api-access-v4lk7\") pod \"maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:39.000994 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:39.000841    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46da344-7402-4b5b-a6ce-197c381f10a7-maas-api-tls\") pod \"maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:39.001487 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:39.001467    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46da344-7402-4b5b-a6ce-197c381f10a7-openshift-service-ca\") pod \"maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:39.003194 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:39.003167    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46da344-7402-4b5b-a6ce-197c381f10a7-maas-api-tls\") pod \"maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:39.009145 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:39.009123    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lk7\" (UniqueName: \"kubernetes.io/projected/f46da344-7402-4b5b-a6ce-197c381f10a7-kube-api-access-v4lk7\") pod \"maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:39.080236 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:39.080160    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:39.206393 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:39.206313    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"]
Jul 20 19:52:39.208771 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:52:39.208744    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46da344_7402_4b5b_a6ce_197c381f10a7.slice/crio-a9c78cf86f2351839eb38aef97d2a7d069b97d62ca2c1c935e00d86c9d2900f6 WatchSource:0}: Error finding container a9c78cf86f2351839eb38aef97d2a7d069b97d62ca2c1c935e00d86c9d2900f6: Status 404 returned error can't find the container with id a9c78cf86f2351839eb38aef97d2a7d069b97d62ca2c1c935e00d86c9d2900f6
Jul 20 19:52:39.210005 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:39.209984    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 19:52:39.778504 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:39.778468    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z" event={"ID":"f46da344-7402-4b5b-a6ce-197c381f10a7","Type":"ContainerStarted","Data":"a9c78cf86f2351839eb38aef97d2a7d069b97d62ca2c1c935e00d86c9d2900f6"}
Jul 20 19:52:40.783834 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:40.783801    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z" event={"ID":"f46da344-7402-4b5b-a6ce-197c381f10a7","Type":"ContainerStarted","Data":"45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da"}
Jul 20 19:52:40.784291 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:40.783939    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:52:40.799325 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:40.799283    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z" podStartSLOduration=2.265159938 podStartE2EDuration="2.799270093s" totalImagesPullingTime="534.110155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:52:40.797498979 +0000 UTC m=+1812.883449986" watchObservedRunningTime="2026-07-20 19:52:40.799270093 +0000 UTC m=+1812.885221081"
Jul 20 19:52:46.793410 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:52:46.793383    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:53:14.669769 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.669729    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"]
Jul 20 19:53:14.670645 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.669937    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z" podUID="f46da344-7402-4b5b-a6ce-197c381f10a7" containerName="maas-api" containerID="cri-o://45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da" gracePeriod=30
Jul 20 19:53:14.911578 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.911554    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:53:14.925736 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.925680    2550 generic.go:356] "Generic (PLEG): container finished" podID="f46da344-7402-4b5b-a6ce-197c381f10a7" containerID="45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da" exitCode=0
Jul 20 19:53:14.925736 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.925715    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"
Jul 20 19:53:14.925858 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.925746    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z" event={"ID":"f46da344-7402-4b5b-a6ce-197c381f10a7","Type":"ContainerDied","Data":"45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da"}
Jul 20 19:53:14.925858 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.925788    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z" event={"ID":"f46da344-7402-4b5b-a6ce-197c381f10a7","Type":"ContainerDied","Data":"a9c78cf86f2351839eb38aef97d2a7d069b97d62ca2c1c935e00d86c9d2900f6"}
Jul 20 19:53:14.925858 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.925810    2550 scope.go:117] "RemoveContainer" containerID="45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da"
Jul 20 19:53:14.934837 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.934809    2550 scope.go:117] "RemoveContainer" containerID="45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da"
Jul 20 19:53:14.935136 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:53:14.935113    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da\": container with ID starting with 45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da not found: ID does not exist" containerID="45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da"
Jul 20 19:53:14.935211 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.935141    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da"} err="failed to get container status \"45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da\": rpc error: code = NotFound desc = could not find container \"45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da\": container with ID starting with 45687283286a329d8550cd137459dd2d5e31485dfb70c12ee39c98a4040931da not found: ID does not exist"
Jul 20 19:53:14.995536 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.995512    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46da344-7402-4b5b-a6ce-197c381f10a7-maas-api-tls\") pod \"f46da344-7402-4b5b-a6ce-197c381f10a7\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") "
Jul 20 19:53:14.995662 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.995572    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46da344-7402-4b5b-a6ce-197c381f10a7-openshift-service-ca\") pod \"f46da344-7402-4b5b-a6ce-197c381f10a7\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") "
Jul 20 19:53:14.995662 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.995619    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4lk7\" (UniqueName: \"kubernetes.io/projected/f46da344-7402-4b5b-a6ce-197c381f10a7-kube-api-access-v4lk7\") pod \"f46da344-7402-4b5b-a6ce-197c381f10a7\" (UID: \"f46da344-7402-4b5b-a6ce-197c381f10a7\") "
Jul 20 19:53:14.995930 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.995895    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46da344-7402-4b5b-a6ce-197c381f10a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f46da344-7402-4b5b-a6ce-197c381f10a7" (UID: "f46da344-7402-4b5b-a6ce-197c381f10a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:53:14.997701 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.997685    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46da344-7402-4b5b-a6ce-197c381f10a7-kube-api-access-v4lk7" (OuterVolumeSpecName: "kube-api-access-v4lk7") pod "f46da344-7402-4b5b-a6ce-197c381f10a7" (UID: "f46da344-7402-4b5b-a6ce-197c381f10a7"). InnerVolumeSpecName "kube-api-access-v4lk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:53:14.997803 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:14.997783    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46da344-7402-4b5b-a6ce-197c381f10a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f46da344-7402-4b5b-a6ce-197c381f10a7" (UID: "f46da344-7402-4b5b-a6ce-197c381f10a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:53:15.096343 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:15.096312    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4lk7\" (UniqueName: \"kubernetes.io/projected/f46da344-7402-4b5b-a6ce-197c381f10a7-kube-api-access-v4lk7\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:53:15.096343 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:15.096338    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f46da344-7402-4b5b-a6ce-197c381f10a7-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:53:15.096343 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:15.096348    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f46da344-7402-4b5b-a6ce-197c381f10a7-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:53:15.247431 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:15.247398    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"]
Jul 20 19:53:15.253869 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:15.253839    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4d9731fc-86f8cf96bb-7sp6z"]
Jul 20 19:53:16.450485 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:16.450452    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f46da344-7402-4b5b-a6ce-197c381f10a7" path="/var/lib/kubelet/pods/f46da344-7402-4b5b-a6ce-197c381f10a7/volumes"
Jul 20 19:53:20.493535 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.493504    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"]
Jul 20 19:53:20.493902 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.493810    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f46da344-7402-4b5b-a6ce-197c381f10a7" containerName="maas-api"
Jul 20 19:53:20.493902 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.493821    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f46da344-7402-4b5b-a6ce-197c381f10a7" containerName="maas-api"
Jul 20 19:53:20.493902 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.493869    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f46da344-7402-4b5b-a6ce-197c381f10a7" containerName="maas-api"
Jul 20 19:53:20.498151 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.498128    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.500346 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.500325    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 20 19:53:20.500475 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.500358    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-6c09e49f-gw-openshift-default-dockercfg-s8wr9\""
Jul 20 19:53:20.508163 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.508144    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"]
Jul 20 19:53:20.541294 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.541248    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed59589-995c-46e8-90d9-622738b6fbd2-istiod-ca-cert\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.541419 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.541316    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-data\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.541419 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.541352    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-podinfo\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.541419 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.541374    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-certs\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.541516 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.541448    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-credential-socket\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.541516 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.541475    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-envoy\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.541516 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.541508    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-socket\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.541607 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.541532    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-token\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.541607 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.541550    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9mzg\" (UniqueName: \"kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-kube-api-access-f9mzg\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.642574 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.642538    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-credential-socket\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.642574 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.642580    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-envoy\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.642795 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.642603    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-socket\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.642795 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.642635    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-token\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.642795 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.642662    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mzg\" (UniqueName: \"kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-kube-api-access-f9mzg\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.642795 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.642688    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed59589-995c-46e8-90d9-622738b6fbd2-istiod-ca-cert\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.642795 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.642720    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-data\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.642795 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.642749    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-podinfo\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.642795 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.642776    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-certs\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.643117 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.643009    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-credential-socket\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.643209 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.643183    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-data\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.643355 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.643183    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-socket\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.643355 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.643238    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-certs\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.643542 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.643523    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed59589-995c-46e8-90d9-622738b6fbd2-istiod-ca-cert\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.645082 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.645063    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-envoy\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.645224 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.645208    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-podinfo\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.651006 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.650985    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-token\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.651322 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.651300    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mzg\" (UniqueName: \"kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-kube-api-access-f9mzg\") pod \"e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") " pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.809027 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.808943    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:20.946445 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:20.946413    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"]
Jul 20 19:53:20.950244 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:53:20.950215    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed59589_995c_46e8_90d9_622738b6fbd2.slice/crio-e95515759947c900aa00ad9b6ea5c3f3941c98762dfe2076a76db4e7875d6d21 WatchSource:0}: Error finding container e95515759947c900aa00ad9b6ea5c3f3941c98762dfe2076a76db4e7875d6d21: Status 404 returned error can't find the container with id e95515759947c900aa00ad9b6ea5c3f3941c98762dfe2076a76db4e7875d6d21
Jul 20 19:53:21.958910 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:21.958862    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59" event={"ID":"4ed59589-995c-46e8-90d9-622738b6fbd2","Type":"ContainerStarted","Data":"e95515759947c900aa00ad9b6ea5c3f3941c98762dfe2076a76db4e7875d6d21"}
Jul 20 19:53:23.496868 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:23.496827    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:53:23.497122 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:23.496907    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:53:23.497122 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:23.496937    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:53:23.969761 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:23.969730    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59" event={"ID":"4ed59589-995c-46e8-90d9-622738b6fbd2","Type":"ContainerStarted","Data":"423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8"}
Jul 20 19:53:23.992251 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:23.992203    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59" podStartSLOduration=1.447889147 podStartE2EDuration="3.992188199s" totalImagesPullingTime="2.544299052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:53:23.99099285 +0000 UTC m=+1856.076943847" watchObservedRunningTime="2026-07-20 19:53:23.992188199 +0000 UTC m=+1856.078139195"
Jul 20 19:53:24.809089 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:24.809051    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:24.813763 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:24.813734    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:24.973805 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:24.973774    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:24.974609 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:24.974592    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:25.682823 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.682788    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"]
Jul 20 19:53:25.686070 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.686053    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:25.691870 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.691835    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6c09e49f-serving-cert\""
Jul 20 19:53:25.705246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.705214    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"]
Jul 20 19:53:25.784963 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.784929    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/318d7760-f726-446a-8b10-a8c94b11428f-openshift-service-ca\") pod \"maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:25.785113 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.784969    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4gt\" (UniqueName: \"kubernetes.io/projected/318d7760-f726-446a-8b10-a8c94b11428f-kube-api-access-4j4gt\") pod \"maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:25.785113 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.785000    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318d7760-f726-446a-8b10-a8c94b11428f-maas-api-tls\") pod \"maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:25.886380 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.886346    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318d7760-f726-446a-8b10-a8c94b11428f-maas-api-tls\") pod \"maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:25.886826 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.886421    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/318d7760-f726-446a-8b10-a8c94b11428f-openshift-service-ca\") pod \"maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:25.886826 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.886493    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4gt\" (UniqueName: \"kubernetes.io/projected/318d7760-f726-446a-8b10-a8c94b11428f-kube-api-access-4j4gt\") pod \"maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:25.886826 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:53:25.886499    2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-serving-cert: secret "maas-api-e2e-migrate-6c09e49f-serving-cert" not found
Jul 20 19:53:25.886826 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:53:25.886567    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/318d7760-f726-446a-8b10-a8c94b11428f-maas-api-tls podName:318d7760-f726-446a-8b10-a8c94b11428f nodeName:}" failed. No retries permitted until 2026-07-20 19:53:26.3865475 +0000 UTC m=+1858.472498476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/318d7760-f726-446a-8b10-a8c94b11428f-maas-api-tls") pod "maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk" (UID: "318d7760-f726-446a-8b10-a8c94b11428f") : secret "maas-api-e2e-migrate-6c09e49f-serving-cert" not found
Jul 20 19:53:25.887117 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.887099    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/318d7760-f726-446a-8b10-a8c94b11428f-openshift-service-ca\") pod \"maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:25.904654 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:25.904628    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4gt\" (UniqueName: \"kubernetes.io/projected/318d7760-f726-446a-8b10-a8c94b11428f-kube-api-access-4j4gt\") pod \"maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:26.389812 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:26.389777    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318d7760-f726-446a-8b10-a8c94b11428f-maas-api-tls\") pod \"maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:26.392149 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:26.392118    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318d7760-f726-446a-8b10-a8c94b11428f-maas-api-tls\") pod \"maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:26.595164 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:26.595128    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:26.726080 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:26.726043    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"]
Jul 20 19:53:26.730019 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:53:26.729984    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318d7760_f726_446a_8b10_a8c94b11428f.slice/crio-eb726d00937e2aa39e5d6667b5aab06c122563a0bd6eca2360bf97eeb391b0e8 WatchSource:0}: Error finding container eb726d00937e2aa39e5d6667b5aab06c122563a0bd6eca2360bf97eeb391b0e8: Status 404 returned error can't find the container with id eb726d00937e2aa39e5d6667b5aab06c122563a0bd6eca2360bf97eeb391b0e8
Jul 20 19:53:26.983732 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:26.983642    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk" event={"ID":"318d7760-f726-446a-8b10-a8c94b11428f","Type":"ContainerStarted","Data":"eb726d00937e2aa39e5d6667b5aab06c122563a0bd6eca2360bf97eeb391b0e8"}
Jul 20 19:53:27.994978 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:27.994941    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk" event={"ID":"318d7760-f726-446a-8b10-a8c94b11428f","Type":"ContainerStarted","Data":"12c46c779554cc509dccabd5a3ec45f012a3a0c0f95e0e646be6351b5dda8f7c"}
Jul 20 19:53:27.995484 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:27.995087    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:28.014205 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:28.014154    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk" podStartSLOduration=2.5190701840000003 podStartE2EDuration="3.01414067s" totalImagesPullingTime="495.070486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:53:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:53:28.0125809 +0000 UTC m=+1860.098531900" watchObservedRunningTime="2026-07-20 19:53:28.01414067 +0000 UTC m=+1860.100091735"
Jul 20 19:53:34.004501 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:34.004473    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:53:37.059405 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:37.059307    2550 generic.go:356] "Generic (PLEG): container finished" podID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerID="423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8" exitCode=0
Jul 20 19:53:37.059405 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:37.059384    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59" event={"ID":"4ed59589-995c-46e8-90d9-622738b6fbd2","Type":"ContainerDied","Data":"423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8"}
Jul 20 19:53:37.059835 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:37.059744    2550 scope.go:117] "RemoveContainer" containerID="423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8"
Jul 20 19:53:37.060780 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:37.060741    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:53:37.060901 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:37.060815    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:53:37.060901 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:37.060844    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:53:38.065601 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:38.065568    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59" event={"ID":"4ed59589-995c-46e8-90d9-622738b6fbd2","Type":"ContainerStarted","Data":"5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41"}
Jul 20 19:53:38.809136 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:38.809103    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:38.810379 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:38.810351    2550 patch_prober.go:28] interesting pod/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 20 19:53:38.810521 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:38.810412    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59" podUID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 20 19:53:39.809962 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:39.809925    2550 patch_prober.go:28] interesting pod/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 20 19:53:39.810383 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:39.809979    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59" podUID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 20 19:53:39.974169 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:39.974115    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:40.809566 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:40.809531    2550 patch_prober.go:28] interesting pod/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 20 19:53:40.809746 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:40.809587    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59" podUID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 20 19:53:41.812864 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:41.812837    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:53:41.813909 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:53:41.813887    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:54:01.050536 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.050455    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"]
Jul 20 19:54:01.051072 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.050714    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk" podUID="318d7760-f726-446a-8b10-a8c94b11428f" containerName="maas-api" containerID="cri-o://12c46c779554cc509dccabd5a3ec45f012a3a0c0f95e0e646be6351b5dda8f7c" gracePeriod=30
Jul 20 19:54:01.167332 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.167244    2550 generic.go:356] "Generic (PLEG): container finished" podID="318d7760-f726-446a-8b10-a8c94b11428f" containerID="12c46c779554cc509dccabd5a3ec45f012a3a0c0f95e0e646be6351b5dda8f7c" exitCode=0
Jul 20 19:54:01.167332 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.167288    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk" event={"ID":"318d7760-f726-446a-8b10-a8c94b11428f","Type":"ContainerDied","Data":"12c46c779554cc509dccabd5a3ec45f012a3a0c0f95e0e646be6351b5dda8f7c"}
Jul 20 19:54:01.292329 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.292307    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:54:01.389731 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.389704    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318d7760-f726-446a-8b10-a8c94b11428f-maas-api-tls\") pod \"318d7760-f726-446a-8b10-a8c94b11428f\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") "
Jul 20 19:54:01.389877 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.389755    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/318d7760-f726-446a-8b10-a8c94b11428f-openshift-service-ca\") pod \"318d7760-f726-446a-8b10-a8c94b11428f\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") "
Jul 20 19:54:01.389877 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.389824    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j4gt\" (UniqueName: \"kubernetes.io/projected/318d7760-f726-446a-8b10-a8c94b11428f-kube-api-access-4j4gt\") pod \"318d7760-f726-446a-8b10-a8c94b11428f\" (UID: \"318d7760-f726-446a-8b10-a8c94b11428f\") "
Jul 20 19:54:01.390107 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.390078    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318d7760-f726-446a-8b10-a8c94b11428f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "318d7760-f726-446a-8b10-a8c94b11428f" (UID: "318d7760-f726-446a-8b10-a8c94b11428f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:54:01.391835 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.391803    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318d7760-f726-446a-8b10-a8c94b11428f-kube-api-access-4j4gt" (OuterVolumeSpecName: "kube-api-access-4j4gt") pod "318d7760-f726-446a-8b10-a8c94b11428f" (UID: "318d7760-f726-446a-8b10-a8c94b11428f"). InnerVolumeSpecName "kube-api-access-4j4gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:54:01.391941 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.391863    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/318d7760-f726-446a-8b10-a8c94b11428f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "318d7760-f726-446a-8b10-a8c94b11428f" (UID: "318d7760-f726-446a-8b10-a8c94b11428f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:54:01.491013 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.490976    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/318d7760-f726-446a-8b10-a8c94b11428f-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:01.491013 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.491009    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/318d7760-f726-446a-8b10-a8c94b11428f-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:01.491186 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:01.491022    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j4gt\" (UniqueName: \"kubernetes.io/projected/318d7760-f726-446a-8b10-a8c94b11428f-kube-api-access-4j4gt\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:02.173188 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:02.173157    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk" event={"ID":"318d7760-f726-446a-8b10-a8c94b11428f","Type":"ContainerDied","Data":"eb726d00937e2aa39e5d6667b5aab06c122563a0bd6eca2360bf97eeb391b0e8"}
Jul 20 19:54:02.173188 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:02.173188    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"
Jul 20 19:54:02.173662 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:02.173218    2550 scope.go:117] "RemoveContainer" containerID="12c46c779554cc509dccabd5a3ec45f012a3a0c0f95e0e646be6351b5dda8f7c"
Jul 20 19:54:02.194968 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:02.194944    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"]
Jul 20 19:54:02.200605 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:02.200583    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6c09e49f-67944c9fd-dz8fk"]
Jul 20 19:54:02.450130 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:02.450049    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="318d7760-f726-446a-8b10-a8c94b11428f" path="/var/lib/kubelet/pods/318d7760-f726-446a-8b10-a8c94b11428f/volumes"
Jul 20 19:54:05.733902 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:05.733863    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"]
Jul 20 19:54:05.734291 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:05.734095    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59" podUID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerName="istio-proxy" containerID="cri-o://5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41" gracePeriod=30
Jul 20 19:54:06.514794 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.514760    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"]
Jul 20 19:54:06.515092 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.515080    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="318d7760-f726-446a-8b10-a8c94b11428f" containerName="maas-api"
Jul 20 19:54:06.515136 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.515094    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="318d7760-f726-446a-8b10-a8c94b11428f" containerName="maas-api"
Jul 20 19:54:06.515169 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.515155    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="318d7760-f726-446a-8b10-a8c94b11428f" containerName="maas-api"
Jul 20 19:54:06.519229 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.519209    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.521401 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.521377    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d9b02c4d-openshift-default-dockercfg-9c677\""
Jul 20 19:54:06.530566 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.530541    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"]
Jul 20 19:54:06.532372 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.532346    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-credential-socket\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.532475 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.532386    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-data\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.532475 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.532436    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-socket\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.532475 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.532457    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-envoy\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.532625 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.532503    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrxl\" (UniqueName: \"kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-kube-api-access-rhrxl\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.532625 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.532536    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-certs\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.532625 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.532601    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-podinfo\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.532768 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.532653    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-token\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.532768 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.532669    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istiod-ca-cert\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.633809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.633772    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-token\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.633809 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.633808    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istiod-ca-cert\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.633838    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-credential-socket\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.633856    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-data\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.633886    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-socket\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.633908    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-envoy\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.633938    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrxl\" (UniqueName: \"kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-kube-api-access-rhrxl\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.633965    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-certs\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.634012    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-podinfo\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634403 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.634301    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-data\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634403 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.634352    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-credential-socket\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634534 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.634515    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-certs\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634589 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.634568    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-socket\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.634626 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.634593    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istiod-ca-cert\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.636290 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.636251    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-envoy\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.636450 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.636434    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-podinfo\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.642152 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.642132    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-token\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.642662 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.642640    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrxl\" (UniqueName: \"kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-kube-api-access-rhrxl\") pod \"e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") " pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.829325 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.829238    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:06.963125 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.963024    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"]
Jul 20 19:54:06.965726 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:54:06.965701    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e0876f_26e5_4f8c_a847_d0ef78e5fdc4.slice/crio-fefba73ffa75a7225e2cc93f75551eebb1d1dc5850599242ff04e045e247b7f4 WatchSource:0}: Error finding container fefba73ffa75a7225e2cc93f75551eebb1d1dc5850599242ff04e045e247b7f4: Status 404 returned error can't find the container with id fefba73ffa75a7225e2cc93f75551eebb1d1dc5850599242ff04e045e247b7f4
Jul 20 19:54:06.967808 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.967775    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:54:06.967931 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.967849    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:54:06.967931 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:06.967897    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:54:07.197108 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:07.197070    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" event={"ID":"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4","Type":"ContainerStarted","Data":"118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c"}
Jul 20 19:54:07.197324 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:07.197122    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" event={"ID":"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4","Type":"ContainerStarted","Data":"fefba73ffa75a7225e2cc93f75551eebb1d1dc5850599242ff04e045e247b7f4"}
Jul 20 19:54:07.219395 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:07.219339    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" podStartSLOduration=1.2193180240000001 podStartE2EDuration="1.219318024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:54:07.218353563 +0000 UTC m=+1899.304304561" watchObservedRunningTime="2026-07-20 19:54:07.219318024 +0000 UTC m=+1899.305269022"
Jul 20 19:54:07.830171 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:07.830132    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:08.768669 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.768635    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"]
Jul 20 19:54:08.771907 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.771889    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:08.774970 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.774945    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d9b02c4d-serving-cert\""
Jul 20 19:54:08.796339 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.796290    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"]
Jul 20 19:54:08.835049 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.835018    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:08.852791 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.852763    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqb9k\" (UniqueName: \"kubernetes.io/projected/aff8c139-631b-4f45-8f60-73dd06e5feda-kube-api-access-bqb9k\") pod \"maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:08.852950 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.852832    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff8c139-631b-4f45-8f60-73dd06e5feda-openshift-service-ca\") pod \"maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:08.852950 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.852866    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff8c139-631b-4f45-8f60-73dd06e5feda-maas-api-tls\") pod \"maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:08.954143 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.954112    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqb9k\" (UniqueName: \"kubernetes.io/projected/aff8c139-631b-4f45-8f60-73dd06e5feda-kube-api-access-bqb9k\") pod \"maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:08.954341 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.954174    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff8c139-631b-4f45-8f60-73dd06e5feda-openshift-service-ca\") pod \"maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:08.954341 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.954209    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff8c139-631b-4f45-8f60-73dd06e5feda-maas-api-tls\") pod \"maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:08.954429 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:54:08.954354    2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-serving-cert: secret "maas-api-e2e-ait-d9b02c4d-serving-cert" not found
Jul 20 19:54:08.954429 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:54:08.954410    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff8c139-631b-4f45-8f60-73dd06e5feda-maas-api-tls podName:aff8c139-631b-4f45-8f60-73dd06e5feda nodeName:}" failed. No retries permitted until 2026-07-20 19:54:09.454391132 +0000 UTC m=+1901.540342111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aff8c139-631b-4f45-8f60-73dd06e5feda-maas-api-tls") pod "maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b" (UID: "aff8c139-631b-4f45-8f60-73dd06e5feda") : secret "maas-api-e2e-ait-d9b02c4d-serving-cert" not found
Jul 20 19:54:08.954799 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.954780    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff8c139-631b-4f45-8f60-73dd06e5feda-openshift-service-ca\") pod \"maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:08.968035 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:08.968014    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqb9k\" (UniqueName: \"kubernetes.io/projected/aff8c139-631b-4f45-8f60-73dd06e5feda-kube-api-access-bqb9k\") pod \"maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:09.205770 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:09.205731    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:09.206782 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:09.206758    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:09.458787 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:09.458705    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff8c139-631b-4f45-8f60-73dd06e5feda-maas-api-tls\") pod \"maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:09.461076 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:09.461051    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff8c139-631b-4f45-8f60-73dd06e5feda-maas-api-tls\") pod \"maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:09.681810 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:09.681771    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:09.818659 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:09.818630    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"]
Jul 20 19:54:09.820814 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:54:09.820784    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff8c139_631b_4f45_8f60_73dd06e5feda.slice/crio-e5b1bfc0a7b57939db9c9587407878d647e4b2157bcbe0298d38e0144f3e8845 WatchSource:0}: Error finding container e5b1bfc0a7b57939db9c9587407878d647e4b2157bcbe0298d38e0144f3e8845: Status 404 returned error can't find the container with id e5b1bfc0a7b57939db9c9587407878d647e4b2157bcbe0298d38e0144f3e8845
Jul 20 19:54:10.210175 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:10.210145    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b" event={"ID":"aff8c139-631b-4f45-8f60-73dd06e5feda","Type":"ContainerStarted","Data":"e5b1bfc0a7b57939db9c9587407878d647e4b2157bcbe0298d38e0144f3e8845"}
Jul 20 19:54:10.977499 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:10.977471    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:54:11.072342 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072190    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-podinfo\") pod \"4ed59589-995c-46e8-90d9-622738b6fbd2\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") "
Jul 20 19:54:11.072342 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072232    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-data\") pod \"4ed59589-995c-46e8-90d9-622738b6fbd2\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") "
Jul 20 19:54:11.072342 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072247    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-credential-socket\") pod \"4ed59589-995c-46e8-90d9-622738b6fbd2\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") "
Jul 20 19:54:11.072342 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072296    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed59589-995c-46e8-90d9-622738b6fbd2-istiod-ca-cert\") pod \"4ed59589-995c-46e8-90d9-622738b6fbd2\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") "
Jul 20 19:54:11.072663 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072365    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-envoy\") pod \"4ed59589-995c-46e8-90d9-622738b6fbd2\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") "
Jul 20 19:54:11.072663 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072391    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-socket\") pod \"4ed59589-995c-46e8-90d9-622738b6fbd2\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") "
Jul 20 19:54:11.072663 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072418    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-token\") pod \"4ed59589-995c-46e8-90d9-622738b6fbd2\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") "
Jul 20 19:54:11.072663 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072477    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9mzg\" (UniqueName: \"kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-kube-api-access-f9mzg\") pod \"4ed59589-995c-46e8-90d9-622738b6fbd2\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") "
Jul 20 19:54:11.072663 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072529    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-certs\") pod \"4ed59589-995c-46e8-90d9-622738b6fbd2\" (UID: \"4ed59589-995c-46e8-90d9-622738b6fbd2\") "
Jul 20 19:54:11.072960 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072701    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ed59589-995c-46e8-90d9-622738b6fbd2" (UID: "4ed59589-995c-46e8-90d9-622738b6fbd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:54:11.072960 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072869    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-credential-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:11.072960 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.072929    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ed59589-995c-46e8-90d9-622738b6fbd2" (UID: "4ed59589-995c-46e8-90d9-622738b6fbd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:54:11.073367 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.073038    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ed59589-995c-46e8-90d9-622738b6fbd2" (UID: "4ed59589-995c-46e8-90d9-622738b6fbd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:54:11.073367 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.073101    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed59589-995c-46e8-90d9-622738b6fbd2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ed59589-995c-46e8-90d9-622738b6fbd2" (UID: "4ed59589-995c-46e8-90d9-622738b6fbd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:54:11.073367 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.073143    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ed59589-995c-46e8-90d9-622738b6fbd2" (UID: "4ed59589-995c-46e8-90d9-622738b6fbd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:54:11.074843 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.074818    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ed59589-995c-46e8-90d9-622738b6fbd2" (UID: "4ed59589-995c-46e8-90d9-622738b6fbd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:54:11.074935 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.074868    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ed59589-995c-46e8-90d9-622738b6fbd2" (UID: "4ed59589-995c-46e8-90d9-622738b6fbd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 19:54:11.075023 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.075009    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-kube-api-access-f9mzg" (OuterVolumeSpecName: "kube-api-access-f9mzg") pod "4ed59589-995c-46e8-90d9-622738b6fbd2" (UID: "4ed59589-995c-46e8-90d9-622738b6fbd2"). InnerVolumeSpecName "kube-api-access-f9mzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:54:11.075104 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.075087    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ed59589-995c-46e8-90d9-622738b6fbd2" (UID: "4ed59589-995c-46e8-90d9-622738b6fbd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:54:11.173703 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.173669    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-data\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:11.173703 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.173697    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ed59589-995c-46e8-90d9-622738b6fbd2-istiod-ca-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:11.173703 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.173708    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-envoy\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:11.173937 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.173716    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:11.173937 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.173725    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-token\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:11.173937 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.173733    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9mzg\" (UniqueName: \"kubernetes.io/projected/4ed59589-995c-46e8-90d9-622738b6fbd2-kube-api-access-f9mzg\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:11.173937 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.173749    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ed59589-995c-46e8-90d9-622738b6fbd2-workload-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:11.173937 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.173759    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ed59589-995c-46e8-90d9-622738b6fbd2-istio-podinfo\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:11.215808 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.215766    2550 generic.go:356] "Generic (PLEG): container finished" podID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerID="5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41" exitCode=0
Jul 20 19:54:11.216248 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.215814    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"
Jul 20 19:54:11.216248 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.215855    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59" event={"ID":"4ed59589-995c-46e8-90d9-622738b6fbd2","Type":"ContainerDied","Data":"5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41"}
Jul 20 19:54:11.216248 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.215917    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59" event={"ID":"4ed59589-995c-46e8-90d9-622738b6fbd2","Type":"ContainerDied","Data":"e95515759947c900aa00ad9b6ea5c3f3941c98762dfe2076a76db4e7875d6d21"}
Jul 20 19:54:11.216248 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.215954    2550 scope.go:117] "RemoveContainer" containerID="5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41"
Jul 20 19:54:11.217594 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.217567    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b" event={"ID":"aff8c139-631b-4f45-8f60-73dd06e5feda","Type":"ContainerStarted","Data":"c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561"}
Jul 20 19:54:11.217862 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.217838    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:11.227377 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.227237    2550 scope.go:117] "RemoveContainer" containerID="423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8"
Jul 20 19:54:11.234448 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.234425    2550 scope.go:117] "RemoveContainer" containerID="5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41"
Jul 20 19:54:11.234685 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:54:11.234668    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41\": container with ID starting with 5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41 not found: ID does not exist" containerID="5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41"
Jul 20 19:54:11.234739 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.234693    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41"} err="failed to get container status \"5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41\": rpc error: code = NotFound desc = could not find container \"5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41\": container with ID starting with 5950911fe34e6d0685404fe7f0b2000436788a0fff89ef40009dde553e36fc41 not found: ID does not exist"
Jul 20 19:54:11.234739 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.234710    2550 scope.go:117] "RemoveContainer" containerID="423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8"
Jul 20 19:54:11.234924 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:54:11.234906    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8\": container with ID starting with 423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8 not found: ID does not exist" containerID="423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8"
Jul 20 19:54:11.234967 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.234930    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8"} err="failed to get container status \"423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8\": rpc error: code = NotFound desc = could not find container \"423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8\": container with ID starting with 423f8ba4aa122c93b2b4c136b559dc411f6868bb06fff73dc01285fba43fd3a8 not found: ID does not exist"
Jul 20 19:54:11.267880 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.267830    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b" podStartSLOduration=2.889433222 podStartE2EDuration="3.267814468s" totalImagesPullingTime="378.381246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:54:11.265910934 +0000 UTC m=+1903.351861931" watchObservedRunningTime="2026-07-20 19:54:11.267814468 +0000 UTC m=+1903.353765471"
Jul 20 19:54:11.292612 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.292572    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"]
Jul 20 19:54:11.315906 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:11.315879    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-6c09e49f-gw-openshift-default-8495768795-ddm59"]
Jul 20 19:54:12.450316 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:12.450283    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ed59589-995c-46e8-90d9-622738b6fbd2" path="/var/lib/kubelet/pods/4ed59589-995c-46e8-90d9-622738b6fbd2/volumes"
Jul 20 19:54:17.234660 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:17.234626    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:23.348513 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:54:23.348471    2550 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e0876f_26e5_4f8c_a847_d0ef78e5fdc4.slice/crio-conmon-118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c.scope\": RecentStats: unable to find data in memory cache]"
Jul 20 19:54:24.206948 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:24.206910    2550 patch_prober.go:28] interesting pod/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 20 19:54:24.207121 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:24.206969    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 20 19:54:24.274970 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:24.274939    2550 generic.go:356] "Generic (PLEG): container finished" podID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerID="118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c" exitCode=0
Jul 20 19:54:24.275136 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:24.275013    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" event={"ID":"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4","Type":"ContainerDied","Data":"118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c"}
Jul 20 19:54:24.275524 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:24.275509    2550 scope.go:117] "RemoveContainer" containerID="118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c"
Jul 20 19:54:24.276458 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:24.276432    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:54:24.276528 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:24.276515    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:54:24.276563 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:24.276549    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:54:25.280897 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:25.280862    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" event={"ID":"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4","Type":"ContainerStarted","Data":"ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956"}
Jul 20 19:54:25.830710 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:25.830666    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:25.830927 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:25.830801    2550 patch_prober.go:28] interesting pod/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 20 19:54:25.830927 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:25.830844    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 20 19:54:26.830439 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:26.830390    2550 patch_prober.go:28] interesting pod/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 20 19:54:26.831425 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:26.830990    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 20 19:54:27.830556 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:27.830508    2550 patch_prober.go:28] interesting pod/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 20 19:54:27.831064 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:27.830594    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 20 19:54:28.833370 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:28.833270    2550 patch_prober.go:28] interesting pod/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Jul 20 19:54:28.833944 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:28.833888    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Jul 20 19:54:29.833952 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:29.833924    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:29.834458 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:29.834353    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:29.834943 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:29.834926    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:44.940238 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:44.940201    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"]
Jul 20 19:54:44.941251 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:44.941203    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b" podUID="aff8c139-631b-4f45-8f60-73dd06e5feda" containerName="maas-api" containerID="cri-o://c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561" gracePeriod=30
Jul 20 19:54:45.178207 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.178181    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:45.280040 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.279949    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff8c139-631b-4f45-8f60-73dd06e5feda-openshift-service-ca\") pod \"aff8c139-631b-4f45-8f60-73dd06e5feda\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") "
Jul 20 19:54:45.280040 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.280030    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff8c139-631b-4f45-8f60-73dd06e5feda-maas-api-tls\") pod \"aff8c139-631b-4f45-8f60-73dd06e5feda\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") "
Jul 20 19:54:45.280284 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.280085    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqb9k\" (UniqueName: \"kubernetes.io/projected/aff8c139-631b-4f45-8f60-73dd06e5feda-kube-api-access-bqb9k\") pod \"aff8c139-631b-4f45-8f60-73dd06e5feda\" (UID: \"aff8c139-631b-4f45-8f60-73dd06e5feda\") "
Jul 20 19:54:45.280381 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.280356    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff8c139-631b-4f45-8f60-73dd06e5feda-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aff8c139-631b-4f45-8f60-73dd06e5feda" (UID: "aff8c139-631b-4f45-8f60-73dd06e5feda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:54:45.282141 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.282108    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff8c139-631b-4f45-8f60-73dd06e5feda-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aff8c139-631b-4f45-8f60-73dd06e5feda" (UID: "aff8c139-631b-4f45-8f60-73dd06e5feda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:54:45.282251 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.282164    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff8c139-631b-4f45-8f60-73dd06e5feda-kube-api-access-bqb9k" (OuterVolumeSpecName: "kube-api-access-bqb9k") pod "aff8c139-631b-4f45-8f60-73dd06e5feda" (UID: "aff8c139-631b-4f45-8f60-73dd06e5feda"). InnerVolumeSpecName "kube-api-access-bqb9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:54:45.369583 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.369549    2550 generic.go:356] "Generic (PLEG): container finished" podID="aff8c139-631b-4f45-8f60-73dd06e5feda" containerID="c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561" exitCode=0
Jul 20 19:54:45.369736 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.369595    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"
Jul 20 19:54:45.369736 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.369634    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b" event={"ID":"aff8c139-631b-4f45-8f60-73dd06e5feda","Type":"ContainerDied","Data":"c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561"}
Jul 20 19:54:45.369736 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.369684    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b" event={"ID":"aff8c139-631b-4f45-8f60-73dd06e5feda","Type":"ContainerDied","Data":"e5b1bfc0a7b57939db9c9587407878d647e4b2157bcbe0298d38e0144f3e8845"}
Jul 20 19:54:45.369736 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.369708    2550 scope.go:117] "RemoveContainer" containerID="c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561"
Jul 20 19:54:45.378073 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.378059    2550 scope.go:117] "RemoveContainer" containerID="c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561"
Jul 20 19:54:45.378313 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:54:45.378297    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561\": container with ID starting with c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561 not found: ID does not exist" containerID="c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561"
Jul 20 19:54:45.378361 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.378319    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561"} err="failed to get container status \"c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561\": rpc error: code = NotFound desc = could not find container \"c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561\": container with ID starting with c701dcd6b06982e4394e0eafd3fff9090c99017fe6e94470a2251ea6d1515561 not found: ID does not exist"
Jul 20 19:54:45.381042 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.381026    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqb9k\" (UniqueName: \"kubernetes.io/projected/aff8c139-631b-4f45-8f60-73dd06e5feda-kube-api-access-bqb9k\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:45.381109 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.381042    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff8c139-631b-4f45-8f60-73dd06e5feda-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:45.381109 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.381053    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff8c139-631b-4f45-8f60-73dd06e5feda-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:45.395835 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.395803    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"]
Jul 20 19:54:45.404095 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:45.404071    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d9b02c4d-fdf499b8c-j787b"]
Jul 20 19:54:46.449846 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:46.449813    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aff8c139-631b-4f45-8f60-73dd06e5feda" path="/var/lib/kubelet/pods/aff8c139-631b-4f45-8f60-73dd06e5feda/volumes"
Jul 20 19:54:50.731202 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:50.731160    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"]
Jul 20 19:54:50.731792 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:50.731425    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy" containerID="cri-o://ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956" gracePeriod=30
Jul 20 19:54:51.599641 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.599610    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"]
Jul 20 19:54:51.599983 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.599966    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerName="istio-proxy"
Jul 20 19:54:51.600065 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.599986    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerName="istio-proxy"
Jul 20 19:54:51.600065 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.600004    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff8c139-631b-4f45-8f60-73dd06e5feda" containerName="maas-api"
Jul 20 19:54:51.600065 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.600011    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff8c139-631b-4f45-8f60-73dd06e5feda" containerName="maas-api"
Jul 20 19:54:51.600217 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.600105    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff8c139-631b-4f45-8f60-73dd06e5feda" containerName="maas-api"
Jul 20 19:54:51.600217 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.600120    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerName="istio-proxy"
Jul 20 19:54:51.604390 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.604370    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.607285 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.607247    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8e9637aa-openshift-default-dockercfg-rv2l7\""
Jul 20 19:54:51.615942 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.615918    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"]
Jul 20 19:54:51.628414 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.628387    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-data\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.628509 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.628431    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-socket\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.628557 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.628503    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbxkl\" (UniqueName: \"kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-kube-api-access-vbxkl\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.628618 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.628576    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-certs\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.628618 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.628600    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687205f0-b3e9-4836-b476-f6f092c738de-istio-podinfo\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.628618 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.628615    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687205f0-b3e9-4836-b476-f6f092c738de-istiod-ca-cert\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.628766 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.628685    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-envoy\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.628766 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.628704    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-istio-token\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.628766 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.628725    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-credential-socket\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.729818 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.729783    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-data\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.729818 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.729820    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-socket\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730022 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.729841    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbxkl\" (UniqueName: \"kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-kube-api-access-vbxkl\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730022 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.729900    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-certs\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730022 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.729925    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687205f0-b3e9-4836-b476-f6f092c738de-istio-podinfo\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730022 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.729949    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687205f0-b3e9-4836-b476-f6f092c738de-istiod-ca-cert\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730022 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.730005    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-envoy\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730237 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.730030    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-istio-token\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730237 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.730195    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-credential-socket\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730365 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.730337    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-certs\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730365 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.730355    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-socket\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730561 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.730540    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-credential-socket\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730637 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.730593    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-data\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.730804 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.730785    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687205f0-b3e9-4836-b476-f6f092c738de-istiod-ca-cert\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.732250 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.732224    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-envoy\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.732576 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.732388    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687205f0-b3e9-4836-b476-f6f092c738de-istio-podinfo\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.741621 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.741595    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-istio-token\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.741718 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.741640    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbxkl\" (UniqueName: \"kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-kube-api-access-vbxkl\") pod \"e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") " pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:51.914332 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:51.914043    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:52.049814 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:52.049789    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"]
Jul 20 19:54:52.051482 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:54:52.051456    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687205f0_b3e9_4836_b476_f6f092c738de.slice/crio-58621abc55cf901f6d58beb37dcfff33ffa9173e562c561425dcd0a47ebdaa34 WatchSource:0}: Error finding container 58621abc55cf901f6d58beb37dcfff33ffa9173e562c561425dcd0a47ebdaa34: Status 404 returned error can't find the container with id 58621abc55cf901f6d58beb37dcfff33ffa9173e562c561425dcd0a47ebdaa34
Jul 20 19:54:52.053533 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:52.053499    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:54:52.053598 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:52.053564    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:54:52.053635 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:52.053612    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:54:52.401200 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:52.401166    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" event={"ID":"687205f0-b3e9-4836-b476-f6f092c738de","Type":"ContainerStarted","Data":"7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff"}
Jul 20 19:54:52.401378 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:52.401229    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" event={"ID":"687205f0-b3e9-4836-b476-f6f092c738de","Type":"ContainerStarted","Data":"58621abc55cf901f6d58beb37dcfff33ffa9173e562c561425dcd0a47ebdaa34"}
Jul 20 19:54:52.429037 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:52.428990    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" podStartSLOduration=1.42897351 podStartE2EDuration="1.42897351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:54:52.426623842 +0000 UTC m=+1944.512574838" watchObservedRunningTime="2026-07-20 19:54:52.42897351 +0000 UTC m=+1944.514924506"
Jul 20 19:54:52.914681 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:52.914645    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:53.919940 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:53.919905    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:54.121027 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.120990    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"]
Jul 20 19:54:54.121319 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.121306    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerName="istio-proxy"
Jul 20 19:54:54.121319 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.121320    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerName="istio-proxy"
Jul 20 19:54:54.121414 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.121390    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ed59589-995c-46e8-90d9-622738b6fbd2" containerName="istio-proxy"
Jul 20 19:54:54.124298 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.124282    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:54.126712 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.126688    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8e9637aa-serving-cert\""
Jul 20 19:54:54.138589 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.138564    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"]
Jul 20 19:54:54.152064 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.152042    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bede12b9-1734-4e31-b7c8-705924ee06d7-maas-api-tls\") pod \"maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:54.152184 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.152101    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvz4p\" (UniqueName: \"kubernetes.io/projected/bede12b9-1734-4e31-b7c8-705924ee06d7-kube-api-access-zvz4p\") pod \"maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:54.152184 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.152153    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bede12b9-1734-4e31-b7c8-705924ee06d7-openshift-service-ca\") pod \"maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:54.252823 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.252736    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvz4p\" (UniqueName: \"kubernetes.io/projected/bede12b9-1734-4e31-b7c8-705924ee06d7-kube-api-access-zvz4p\") pod \"maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:54.252823 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.252779    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bede12b9-1734-4e31-b7c8-705924ee06d7-openshift-service-ca\") pod \"maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:54.253009 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.252834    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bede12b9-1734-4e31-b7c8-705924ee06d7-maas-api-tls\") pod \"maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:54.253009 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:54:54.252954    2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-serving-cert: secret "maas-api-e2e-derive-8e9637aa-serving-cert" not found
Jul 20 19:54:54.253080 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:54:54.253033    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bede12b9-1734-4e31-b7c8-705924ee06d7-maas-api-tls podName:bede12b9-1734-4e31-b7c8-705924ee06d7 nodeName:}" failed. No retries permitted until 2026-07-20 19:54:54.753014756 +0000 UTC m=+1946.838965733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bede12b9-1734-4e31-b7c8-705924ee06d7-maas-api-tls") pod "maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8" (UID: "bede12b9-1734-4e31-b7c8-705924ee06d7") : secret "maas-api-e2e-derive-8e9637aa-serving-cert" not found
Jul 20 19:54:54.253546 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.253528    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bede12b9-1734-4e31-b7c8-705924ee06d7-openshift-service-ca\") pod \"maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:54.272605 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.272574    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvz4p\" (UniqueName: \"kubernetes.io/projected/bede12b9-1734-4e31-b7c8-705924ee06d7-kube-api-access-zvz4p\") pod \"maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:54.410278 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.410229    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:54.411095 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.411078    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:54:54.757180 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.757145    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bede12b9-1734-4e31-b7c8-705924ee06d7-maas-api-tls\") pod \"maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:54.759520 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:54.759500    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bede12b9-1734-4e31-b7c8-705924ee06d7-maas-api-tls\") pod \"maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:55.034493 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:55.034405    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:55.173295 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:55.173265    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"]
Jul 20 19:54:55.175718 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:54:55.175683    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbede12b9_1734_4e31_b7c8_705924ee06d7.slice/crio-01b54970219bc2ad7d983cb538b52b7f8b6500dd69bcfb10cadcebc4d5ef9393 WatchSource:0}: Error finding container 01b54970219bc2ad7d983cb538b52b7f8b6500dd69bcfb10cadcebc4d5ef9393: Status 404 returned error can't find the container with id 01b54970219bc2ad7d983cb538b52b7f8b6500dd69bcfb10cadcebc4d5ef9393
Jul 20 19:54:55.415047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:55.415011    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8" event={"ID":"bede12b9-1734-4e31-b7c8-705924ee06d7","Type":"ContainerStarted","Data":"01b54970219bc2ad7d983cb538b52b7f8b6500dd69bcfb10cadcebc4d5ef9393"}
Jul 20 19:54:55.974168 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:55.974145    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:56.069909 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.069820    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-socket\") pod \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") "
Jul 20 19:54:56.070354 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.069866    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-data\") pod \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") "
Jul 20 19:54:56.070354 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070074    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-credential-socket\") pod \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") "
Jul 20 19:54:56.070354 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070103    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhrxl\" (UniqueName: \"kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-kube-api-access-rhrxl\") pod \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") "
Jul 20 19:54:56.070354 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070154    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-certs\") pod \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") "
Jul 20 19:54:56.070354 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070188    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istiod-ca-cert\") pod \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") "
Jul 20 19:54:56.070354 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070222    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-envoy\") pod \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") "
Jul 20 19:54:56.070354 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070243    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-token\") pod \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") "
Jul 20 19:54:56.070354 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070321    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-podinfo\") pod \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\" (UID: \"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4\") "
Jul 20 19:54:56.070802 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070362    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" (UID: "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:54:56.070802 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070569    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:56.070802 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070373    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" (UID: "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:54:56.070802 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070655    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" (UID: "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:54:56.070802 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.070772    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" (UID: "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:54:56.071518 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.071485    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-data" (OuterVolumeSpecName: "istio-data") pod "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" (UID: "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:54:56.072494 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.072471    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-token" (OuterVolumeSpecName: "istio-token") pod "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" (UID: "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:54:56.072597 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.072570    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" (UID: "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:54:56.072671 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.072654    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-kube-api-access-rhrxl" (OuterVolumeSpecName: "kube-api-access-rhrxl") pod "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" (UID: "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4"). InnerVolumeSpecName "kube-api-access-rhrxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:54:56.072671 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.072664    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" (UID: "c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 19:54:56.171921 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.171881    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-podinfo\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:56.171921 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.171906    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-data\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:56.171921 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.171916    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-credential-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:56.171921 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.171925    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhrxl\" (UniqueName: \"kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-kube-api-access-rhrxl\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:56.171921 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.171933    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-workload-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:56.172224 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.171942    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istiod-ca-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:56.172224 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.171949    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-envoy\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:56.172224 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.171956    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4-istio-token\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:54:56.420941 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.420902    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8" event={"ID":"bede12b9-1734-4e31-b7c8-705924ee06d7","Type":"ContainerStarted","Data":"5da7af0114fa2fafa9680091b6f345e87ea677459e1df093ed9dbf6a067d40f0"}
Jul 20 19:54:56.421114 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.421062    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:54:56.422562 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.422534    2550 generic.go:356] "Generic (PLEG): container finished" podID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerID="ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956" exitCode=0
Jul 20 19:54:56.422684 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.422588    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"
Jul 20 19:54:56.422684 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.422615    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" event={"ID":"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4","Type":"ContainerDied","Data":"ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956"}
Jul 20 19:54:56.422684 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.422656    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8" event={"ID":"c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4","Type":"ContainerDied","Data":"fefba73ffa75a7225e2cc93f75551eebb1d1dc5850599242ff04e045e247b7f4"}
Jul 20 19:54:56.422684 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.422677    2550 scope.go:117] "RemoveContainer" containerID="ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956"
Jul 20 19:54:56.432458 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.432371    2550 scope.go:117] "RemoveContainer" containerID="118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c"
Jul 20 19:54:56.439923 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.439908    2550 scope.go:117] "RemoveContainer" containerID="ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956"
Jul 20 19:54:56.440171 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:54:56.440150    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956\": container with ID starting with ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956 not found: ID does not exist" containerID="ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956"
Jul 20 19:54:56.440289 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.440175    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956"} err="failed to get container status \"ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956\": rpc error: code = NotFound desc = could not find container \"ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956\": container with ID starting with ffdcc2b6a29facd98386db8b29f22308ad6486edbe455ae0e5153a2c922de956 not found: ID does not exist"
Jul 20 19:54:56.440289 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.440193    2550 scope.go:117] "RemoveContainer" containerID="118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c"
Jul 20 19:54:56.440501 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:54:56.440484    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c\": container with ID starting with 118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c not found: ID does not exist" containerID="118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c"
Jul 20 19:54:56.440561 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.440510    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c"} err="failed to get container status \"118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c\": rpc error: code = NotFound desc = could not find container \"118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c\": container with ID starting with 118f13f4b7e3cb6df0cdaddee1809bc672702e4bf151947a229341bb4722e54c not found: ID does not exist"
Jul 20 19:54:56.443042 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.443005    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8" podStartSLOduration=2.0384714 podStartE2EDuration="2.4429938s" totalImagesPullingTime="404.5224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:54:56.441173173 +0000 UTC m=+1948.527124169" watchObservedRunningTime="2026-07-20 19:54:56.4429938 +0000 UTC m=+1948.528944796"
Jul 20 19:54:56.456982 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.456963    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"]
Jul 20 19:54:56.459845 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:56.459824    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d9b02c4d-openshift-default-77698df9bc-nqpp8"]
Jul 20 19:54:58.450847 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:54:58.450810    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" path="/var/lib/kubelet/pods/c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4/volumes"
Jul 20 19:55:02.440944 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:02.440908    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:55:07.474530 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:07.474494    2550 generic.go:356] "Generic (PLEG): container finished" podID="687205f0-b3e9-4836-b476-f6f092c738de" containerID="7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff" exitCode=0
Jul 20 19:55:07.474971 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:07.474571    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" event={"ID":"687205f0-b3e9-4836-b476-f6f092c738de","Type":"ContainerDied","Data":"7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff"}
Jul 20 19:55:07.475099 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:07.475084    2550 scope.go:117] "RemoveContainer" containerID="7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff"
Jul 20 19:55:07.476095 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:07.476009    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:55:07.476095 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:07.476068    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:55:07.476245 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:07.476099    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:55:08.481164 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:08.481136    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" event={"ID":"687205f0-b3e9-4836-b476-f6f092c738de","Type":"ContainerStarted","Data":"84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1"}
Jul 20 19:55:08.915596 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:08.915489    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:55:08.917210 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:08.917178    2550 patch_prober.go:28] interesting pod/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 20 19:55:08.917386 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:08.917242    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" podUID="687205f0-b3e9-4836-b476-f6f092c738de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 20 19:55:09.410396 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:09.410363    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:55:09.916006 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:09.915960    2550 patch_prober.go:28] interesting pod/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 20 19:55:09.916560 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:09.916028    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" podUID="687205f0-b3e9-4836-b476-f6f092c738de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 20 19:55:10.915780 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:10.915734    2550 patch_prober.go:28] interesting pod/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 20 19:55:10.916002 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:10.915800    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" podUID="687205f0-b3e9-4836-b476-f6f092c738de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 20 19:55:11.915475 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:11.915435    2550 patch_prober.go:28] interesting pod/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Jul 20 19:55:11.916116 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:11.916080    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" podUID="687205f0-b3e9-4836-b476-f6f092c738de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Jul 20 19:55:12.919370 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:12.919341    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:55:12.920469 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:12.920449    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:55:29.403375 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.403343    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"]
Jul 20 19:55:29.403852 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.403589    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8" podUID="bede12b9-1734-4e31-b7c8-705924ee06d7" containerName="maas-api" containerID="cri-o://5da7af0114fa2fafa9680091b6f345e87ea677459e1df093ed9dbf6a067d40f0" gracePeriod=30
Jul 20 19:55:29.569351 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.569311    2550 generic.go:356] "Generic (PLEG): container finished" podID="bede12b9-1734-4e31-b7c8-705924ee06d7" containerID="5da7af0114fa2fafa9680091b6f345e87ea677459e1df093ed9dbf6a067d40f0" exitCode=0
Jul 20 19:55:29.569515 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.569397    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8" event={"ID":"bede12b9-1734-4e31-b7c8-705924ee06d7","Type":"ContainerDied","Data":"5da7af0114fa2fafa9680091b6f345e87ea677459e1df093ed9dbf6a067d40f0"}
Jul 20 19:55:29.646226 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.646203    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:55:29.765991 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.765911    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bede12b9-1734-4e31-b7c8-705924ee06d7-maas-api-tls\") pod \"bede12b9-1734-4e31-b7c8-705924ee06d7\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") "
Jul 20 19:55:29.765991 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.765949    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bede12b9-1734-4e31-b7c8-705924ee06d7-openshift-service-ca\") pod \"bede12b9-1734-4e31-b7c8-705924ee06d7\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") "
Jul 20 19:55:29.766178 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.766067    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvz4p\" (UniqueName: \"kubernetes.io/projected/bede12b9-1734-4e31-b7c8-705924ee06d7-kube-api-access-zvz4p\") pod \"bede12b9-1734-4e31-b7c8-705924ee06d7\" (UID: \"bede12b9-1734-4e31-b7c8-705924ee06d7\") "
Jul 20 19:55:29.766329 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.766305    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bede12b9-1734-4e31-b7c8-705924ee06d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bede12b9-1734-4e31-b7c8-705924ee06d7" (UID: "bede12b9-1734-4e31-b7c8-705924ee06d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:55:29.767990 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.767961    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bede12b9-1734-4e31-b7c8-705924ee06d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bede12b9-1734-4e31-b7c8-705924ee06d7" (UID: "bede12b9-1734-4e31-b7c8-705924ee06d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:55:29.768114 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.768055    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bede12b9-1734-4e31-b7c8-705924ee06d7-kube-api-access-zvz4p" (OuterVolumeSpecName: "kube-api-access-zvz4p") pod "bede12b9-1734-4e31-b7c8-705924ee06d7" (UID: "bede12b9-1734-4e31-b7c8-705924ee06d7"). InnerVolumeSpecName "kube-api-access-zvz4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:55:29.866959 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.866922    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bede12b9-1734-4e31-b7c8-705924ee06d7-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:29.866959 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.866951    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bede12b9-1734-4e31-b7c8-705924ee06d7-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:29.866959 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:29.866961    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvz4p\" (UniqueName: \"kubernetes.io/projected/bede12b9-1734-4e31-b7c8-705924ee06d7-kube-api-access-zvz4p\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:30.574609 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:30.574580    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"
Jul 20 19:55:30.575019 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:30.574614    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8" event={"ID":"bede12b9-1734-4e31-b7c8-705924ee06d7","Type":"ContainerDied","Data":"01b54970219bc2ad7d983cb538b52b7f8b6500dd69bcfb10cadcebc4d5ef9393"}
Jul 20 19:55:30.575019 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:30.574670    2550 scope.go:117] "RemoveContainer" containerID="5da7af0114fa2fafa9680091b6f345e87ea677459e1df093ed9dbf6a067d40f0"
Jul 20 19:55:30.591048 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:30.591025    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"]
Jul 20 19:55:30.596753 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:30.596730    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8e9637aa-5cb7f858fd-kmmr8"]
Jul 20 19:55:32.450037 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:32.450004    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bede12b9-1734-4e31-b7c8-705924ee06d7" path="/var/lib/kubelet/pods/bede12b9-1734-4e31-b7c8-705924ee06d7/volumes"
Jul 20 19:55:33.939481 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:33.939448    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"]
Jul 20 19:55:33.939878 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:33.939736    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" podUID="687205f0-b3e9-4836-b476-f6f092c738de" containerName="istio-proxy" containerID="cri-o://84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1" gracePeriod=30
Jul 20 19:55:39.188148 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.188118    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:55:39.249593 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.249516    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-socket\") pod \"687205f0-b3e9-4836-b476-f6f092c738de\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") "
Jul 20 19:55:39.249593 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.249552    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-data\") pod \"687205f0-b3e9-4836-b476-f6f092c738de\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") "
Jul 20 19:55:39.249832 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.249594    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbxkl\" (UniqueName: \"kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-kube-api-access-vbxkl\") pod \"687205f0-b3e9-4836-b476-f6f092c738de\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") "
Jul 20 19:55:39.249832 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.249623    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-istio-token\") pod \"687205f0-b3e9-4836-b476-f6f092c738de\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") "
Jul 20 19:55:39.249832 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.249647    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-credential-socket\") pod \"687205f0-b3e9-4836-b476-f6f092c738de\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") "
Jul 20 19:55:39.249832 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.249719    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-envoy\") pod \"687205f0-b3e9-4836-b476-f6f092c738de\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") "
Jul 20 19:55:39.249832 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.249769    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-certs\") pod \"687205f0-b3e9-4836-b476-f6f092c738de\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") "
Jul 20 19:55:39.249832 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.249798    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687205f0-b3e9-4836-b476-f6f092c738de-istio-podinfo\") pod \"687205f0-b3e9-4836-b476-f6f092c738de\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") "
Jul 20 19:55:39.249832 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.249828    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687205f0-b3e9-4836-b476-f6f092c738de-istiod-ca-cert\") pod \"687205f0-b3e9-4836-b476-f6f092c738de\" (UID: \"687205f0-b3e9-4836-b476-f6f092c738de\") "
Jul 20 19:55:39.250161 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.249835    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "687205f0-b3e9-4836-b476-f6f092c738de" (UID: "687205f0-b3e9-4836-b476-f6f092c738de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:55:39.250161 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.250088    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:39.250317 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.250283    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "687205f0-b3e9-4836-b476-f6f092c738de" (UID: "687205f0-b3e9-4836-b476-f6f092c738de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:55:39.250456 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.250425    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "687205f0-b3e9-4836-b476-f6f092c738de" (UID: "687205f0-b3e9-4836-b476-f6f092c738de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:55:39.250519 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.250446    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687205f0-b3e9-4836-b476-f6f092c738de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "687205f0-b3e9-4836-b476-f6f092c738de" (UID: "687205f0-b3e9-4836-b476-f6f092c738de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:55:39.250519 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.250496    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-data" (OuterVolumeSpecName: "istio-data") pod "687205f0-b3e9-4836-b476-f6f092c738de" (UID: "687205f0-b3e9-4836-b476-f6f092c738de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:55:39.252156 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.252130    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-kube-api-access-vbxkl" (OuterVolumeSpecName: "kube-api-access-vbxkl") pod "687205f0-b3e9-4836-b476-f6f092c738de" (UID: "687205f0-b3e9-4836-b476-f6f092c738de"). InnerVolumeSpecName "kube-api-access-vbxkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:55:39.252359 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.252332    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/687205f0-b3e9-4836-b476-f6f092c738de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "687205f0-b3e9-4836-b476-f6f092c738de" (UID: "687205f0-b3e9-4836-b476-f6f092c738de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 19:55:39.252429 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.252408    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "687205f0-b3e9-4836-b476-f6f092c738de" (UID: "687205f0-b3e9-4836-b476-f6f092c738de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:55:39.252541 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.252522    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-istio-token" (OuterVolumeSpecName: "istio-token") pod "687205f0-b3e9-4836-b476-f6f092c738de" (UID: "687205f0-b3e9-4836-b476-f6f092c738de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:55:39.351080 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.351044    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-workload-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:39.351080 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.351074    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687205f0-b3e9-4836-b476-f6f092c738de-istio-podinfo\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:39.351080 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.351082    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687205f0-b3e9-4836-b476-f6f092c738de-istiod-ca-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:39.351080 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.351090    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-data\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:39.351380 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.351098    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbxkl\" (UniqueName: \"kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-kube-api-access-vbxkl\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:39.351380 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.351108    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687205f0-b3e9-4836-b476-f6f092c738de-istio-token\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:39.351380 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.351116    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-credential-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:39.351380 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.351124    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687205f0-b3e9-4836-b476-f6f092c738de-istio-envoy\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:55:39.615829 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.615756    2550 generic.go:356] "Generic (PLEG): container finished" podID="687205f0-b3e9-4836-b476-f6f092c738de" containerID="84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1" exitCode=0
Jul 20 19:55:39.615829 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.615802    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" event={"ID":"687205f0-b3e9-4836-b476-f6f092c738de","Type":"ContainerDied","Data":"84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1"}
Jul 20 19:55:39.615829 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.615814    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"
Jul 20 19:55:39.616047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.615843    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn" event={"ID":"687205f0-b3e9-4836-b476-f6f092c738de","Type":"ContainerDied","Data":"58621abc55cf901f6d58beb37dcfff33ffa9173e562c561425dcd0a47ebdaa34"}
Jul 20 19:55:39.616047 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.615864    2550 scope.go:117] "RemoveContainer" containerID="84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1"
Jul 20 19:55:39.624627 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.624498    2550 scope.go:117] "RemoveContainer" containerID="7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff"
Jul 20 19:55:39.633321 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.633301    2550 scope.go:117] "RemoveContainer" containerID="84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1"
Jul 20 19:55:39.633619 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:55:39.633599    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1\": container with ID starting with 84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1 not found: ID does not exist" containerID="84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1"
Jul 20 19:55:39.633700 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.633627    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1"} err="failed to get container status \"84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1\": rpc error: code = NotFound desc = could not find container \"84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1\": container with ID starting with 84082cdec546037b909879ac90e4ace41c8a0297e3ac9fc230641c72b84cf8c1 not found: ID does not exist"
Jul 20 19:55:39.633700 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.633644    2550 scope.go:117] "RemoveContainer" containerID="7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff"
Jul 20 19:55:39.633949 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:55:39.633931    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff\": container with ID starting with 7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff not found: ID does not exist" containerID="7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff"
Jul 20 19:55:39.633999 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.633954    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff"} err="failed to get container status \"7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff\": rpc error: code = NotFound desc = could not find container \"7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff\": container with ID starting with 7ba85cf1881e95c8145eb8d235e4ff0720f755a61c84ebafcc117f1ef6b1a4ff not found: ID does not exist"
Jul 20 19:55:39.646190 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.643356    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"]
Jul 20 19:55:39.648223 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:39.648199    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8e9637aa-openshift-default-59598d8f75-4jpqn"]
Jul 20 19:55:40.449464 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:55:40.449431    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687205f0-b3e9-4836-b476-f6f092c738de" path="/var/lib/kubelet/pods/687205f0-b3e9-4836-b476-f6f092c738de/volumes"
Jul 20 19:57:12.394114 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394037    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"]
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394365    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687205f0-b3e9-4836-b476-f6f092c738de" containerName="istio-proxy"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394377    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="687205f0-b3e9-4836-b476-f6f092c738de" containerName="istio-proxy"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394385    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bede12b9-1734-4e31-b7c8-705924ee06d7" containerName="maas-api"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394390    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bede12b9-1734-4e31-b7c8-705924ee06d7" containerName="maas-api"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394396    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394402    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394407    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394412    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394418    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687205f0-b3e9-4836-b476-f6f092c738de" containerName="istio-proxy"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394422    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="687205f0-b3e9-4836-b476-f6f092c738de" containerName="istio-proxy"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394477    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394484    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bede12b9-1734-4e31-b7c8-705924ee06d7" containerName="maas-api"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394489    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="687205f0-b3e9-4836-b476-f6f092c738de" containerName="istio-proxy"
Jul 20 19:57:12.394692 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.394495    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="687205f0-b3e9-4836-b476-f6f092c738de" containerName="istio-proxy"
Jul 20 19:57:12.397161 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.397142    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.399405 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.399382    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fd3c7e9a-openshift-default-dockercfg-vgbcx\""
Jul 20 19:57:12.399524 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.399385    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 20 19:57:12.421100 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.421073    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"]
Jul 20 19:57:12.527474 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.527445    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istiod-ca-cert\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.527474 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.527477    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-podinfo\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.527687 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.527499    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-socket\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.527687 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.527517    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-credential-socket\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.527687 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.527579    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-envoy\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.527687 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.527622    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-data\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.527687 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.527651    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-token\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.527687 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.527666    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-549mx\" (UniqueName: \"kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-kube-api-access-549mx\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.527894 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.527691    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-certs\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629037 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629003    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-certs\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629219 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629064    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istiod-ca-cert\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629219 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629171    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-podinfo\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629219 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629207    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-socket\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629414 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629234    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-credential-socket\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629414 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629289    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-envoy\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629414 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629318    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-data\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629414 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629370    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-token\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629414 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629401    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549mx\" (UniqueName: \"kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-kube-api-access-549mx\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629680 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629552    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-certs\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629860 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629833    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-socket\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629981 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629861    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istiod-ca-cert\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629981 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629883    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-data\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.629981 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.629906    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-credential-socket\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.631514 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.631496    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-envoy\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.631733 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.631715    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-podinfo\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.638620 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.638594    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-token\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.638852 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.638833    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-549mx\" (UniqueName: \"kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-kube-api-access-549mx\") pod \"e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") " pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.706354 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.706232    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:12.838060 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.838023    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"]
Jul 20 19:57:12.840977 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:57:12.840949    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3d3e16_ccdc_4e18_993f_ef60576eabdd.slice/crio-2c6c06cc8e534c06de4dae7ea1503faced79d8daa000207e48354182ad36d07c WatchSource:0}: Error finding container 2c6c06cc8e534c06de4dae7ea1503faced79d8daa000207e48354182ad36d07c: Status 404 returned error can't find the container with id 2c6c06cc8e534c06de4dae7ea1503faced79d8daa000207e48354182ad36d07c
Jul 20 19:57:12.843099 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.843064    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:57:12.843177 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.843135    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:57:12.843177 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:12.843162    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:57:13.017139 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:13.017049    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" event={"ID":"8e3d3e16-ccdc-4e18-993f-ef60576eabdd","Type":"ContainerStarted","Data":"5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018"}
Jul 20 19:57:13.017139 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:13.017130    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" event={"ID":"8e3d3e16-ccdc-4e18-993f-ef60576eabdd","Type":"ContainerStarted","Data":"2c6c06cc8e534c06de4dae7ea1503faced79d8daa000207e48354182ad36d07c"}
Jul 20 19:57:13.037063 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:13.037012    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" podStartSLOduration=1.036997322 podStartE2EDuration="1.036997322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:57:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:57:13.035897571 +0000 UTC m=+2085.121848582" watchObservedRunningTime="2026-07-20 19:57:13.036997322 +0000 UTC m=+2085.122948319"
Jul 20 19:57:13.707081 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:13.707038    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:13.712773 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:13.712745    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:14.022894 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:14.022802    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:14.023751 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:14.023726    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:15.453455 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.453409    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"]
Jul 20 19:57:15.453980 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.453957    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2e0876f-26e5-4f8c-a847-d0ef78e5fdc4" containerName="istio-proxy"
Jul 20 19:57:15.457190 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.457167    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:15.459518 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.459493    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fd3c7e9a-serving-cert\""
Jul 20 19:57:15.476203 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.476178    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"]
Jul 20 19:57:15.557426 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.557353    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6bd3b9-7218-4996-bd6b-78896b1026ed-maas-api-tls\") pod \"maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:15.557628 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.557444    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgk62\" (UniqueName: \"kubernetes.io/projected/6e6bd3b9-7218-4996-bd6b-78896b1026ed-kube-api-access-mgk62\") pod \"maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:15.557628 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.557523    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6bd3b9-7218-4996-bd6b-78896b1026ed-openshift-service-ca\") pod \"maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:15.658629 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.658585    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6bd3b9-7218-4996-bd6b-78896b1026ed-openshift-service-ca\") pod \"maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:15.658833 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.658683    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6bd3b9-7218-4996-bd6b-78896b1026ed-maas-api-tls\") pod \"maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:15.658833 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.658715    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgk62\" (UniqueName: \"kubernetes.io/projected/6e6bd3b9-7218-4996-bd6b-78896b1026ed-kube-api-access-mgk62\") pod \"maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:15.659763 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.659731    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6bd3b9-7218-4996-bd6b-78896b1026ed-openshift-service-ca\") pod \"maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:15.659913 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:57:15.659850    2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-serving-cert: secret "maas-api-e2e-mt-fd3c7e9a-serving-cert" not found
Jul 20 19:57:15.659913 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:57:15.659907    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e6bd3b9-7218-4996-bd6b-78896b1026ed-maas-api-tls podName:6e6bd3b9-7218-4996-bd6b-78896b1026ed nodeName:}" failed. No retries permitted until 2026-07-20 19:57:16.159887926 +0000 UTC m=+2088.245838914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e6bd3b9-7218-4996-bd6b-78896b1026ed-maas-api-tls") pod "maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p" (UID: "6e6bd3b9-7218-4996-bd6b-78896b1026ed") : secret "maas-api-e2e-mt-fd3c7e9a-serving-cert" not found
Jul 20 19:57:15.673815 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:15.673347    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgk62\" (UniqueName: \"kubernetes.io/projected/6e6bd3b9-7218-4996-bd6b-78896b1026ed-kube-api-access-mgk62\") pod \"maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:16.162468 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:16.162432    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6bd3b9-7218-4996-bd6b-78896b1026ed-maas-api-tls\") pod \"maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:16.164861 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:16.164827    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6bd3b9-7218-4996-bd6b-78896b1026ed-maas-api-tls\") pod \"maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:16.367593 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:16.367557    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:16.512420 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:16.512388    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"]
Jul 20 19:57:16.516105 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:57:16.516067    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6bd3b9_7218_4996_bd6b_78896b1026ed.slice/crio-1f3e8f75bbe4e71688cc320906c49580b02c1f7b6b06a834b3dfb9b750ecf79f WatchSource:0}: Error finding container 1f3e8f75bbe4e71688cc320906c49580b02c1f7b6b06a834b3dfb9b750ecf79f: Status 404 returned error can't find the container with id 1f3e8f75bbe4e71688cc320906c49580b02c1f7b6b06a834b3dfb9b750ecf79f
Jul 20 19:57:17.037331 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:17.037292    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p" event={"ID":"6e6bd3b9-7218-4996-bd6b-78896b1026ed","Type":"ContainerStarted","Data":"1f3e8f75bbe4e71688cc320906c49580b02c1f7b6b06a834b3dfb9b750ecf79f"}
Jul 20 19:57:18.043229 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:18.043198    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p" event={"ID":"6e6bd3b9-7218-4996-bd6b-78896b1026ed","Type":"ContainerStarted","Data":"b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd"}
Jul 20 19:57:18.043668 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:18.043332    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:18.060859 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:18.060817    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p" podStartSLOduration=2.515126756 podStartE2EDuration="3.060803205s" totalImagesPullingTime="545.676449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:57:18.060247997 +0000 UTC m=+2090.146199018" watchObservedRunningTime="2026-07-20 19:57:18.060803205 +0000 UTC m=+2090.146754201"
Jul 20 19:57:24.052016 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:24.051984    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:28.470820 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:28.470793    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:57:28.475477 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:28.475455    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 19:57:29.024149 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:29.024117    2550 patch_prober.go:28] interesting pod/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 20 19:57:29.024348 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:29.024182    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 20 19:57:29.089730 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:29.089698    2550 generic.go:356] "Generic (PLEG): container finished" podID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerID="5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018" exitCode=0
Jul 20 19:57:29.089730 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:29.089736    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" event={"ID":"8e3d3e16-ccdc-4e18-993f-ef60576eabdd","Type":"ContainerDied","Data":"5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018"}
Jul 20 19:57:29.090132 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:29.090117    2550 scope.go:117] "RemoveContainer" containerID="5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018"
Jul 20 19:57:29.091107 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:29.091079    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:57:29.091166 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:29.091151    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:57:29.091206 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:29.091197    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:57:30.095928 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:30.095893    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" event={"ID":"8e3d3e16-ccdc-4e18-993f-ef60576eabdd","Type":"ContainerStarted","Data":"5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7"}
Jul 20 19:57:30.706953 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:30.706910    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:30.707133 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:30.707104    2550 patch_prober.go:28] interesting pod/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 20 19:57:30.707178 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:30.707156    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 20 19:57:31.707161 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:31.707115    2550 patch_prober.go:28] interesting pod/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 20 19:57:31.707578 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:31.707210    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 20 19:57:32.707075 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:32.707041    2550 patch_prober.go:28] interesting pod/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 20 19:57:32.707243 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:32.707105    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 20 19:57:33.706635 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:33.706595    2550 patch_prober.go:28] interesting pod/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Jul 20 19:57:33.706819 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:33.706661    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Jul 20 19:57:34.710759 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:34.710729    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:34.711136 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:34.711024    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:34.711685 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:34.711667    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:38.214350 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.214317    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"]
Jul 20 19:57:38.214715 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.214538    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p" podUID="6e6bd3b9-7218-4996-bd6b-78896b1026ed" containerName="maas-api" containerID="cri-o://b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd" gracePeriod=30
Jul 20 19:57:38.458853 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.458829    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:38.542634 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.542547    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6bd3b9-7218-4996-bd6b-78896b1026ed-maas-api-tls\") pod \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") "
Jul 20 19:57:38.542634 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.542597    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6bd3b9-7218-4996-bd6b-78896b1026ed-openshift-service-ca\") pod \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") "
Jul 20 19:57:38.542861 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.542654    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgk62\" (UniqueName: \"kubernetes.io/projected/6e6bd3b9-7218-4996-bd6b-78896b1026ed-kube-api-access-mgk62\") pod \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\" (UID: \"6e6bd3b9-7218-4996-bd6b-78896b1026ed\") "
Jul 20 19:57:38.543012 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.542988    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e6bd3b9-7218-4996-bd6b-78896b1026ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e6bd3b9-7218-4996-bd6b-78896b1026ed" (UID: "6e6bd3b9-7218-4996-bd6b-78896b1026ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:57:38.544720 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.544693    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e6bd3b9-7218-4996-bd6b-78896b1026ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e6bd3b9-7218-4996-bd6b-78896b1026ed" (UID: "6e6bd3b9-7218-4996-bd6b-78896b1026ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:57:38.544818 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.544714    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6bd3b9-7218-4996-bd6b-78896b1026ed-kube-api-access-mgk62" (OuterVolumeSpecName: "kube-api-access-mgk62") pod "6e6bd3b9-7218-4996-bd6b-78896b1026ed" (UID: "6e6bd3b9-7218-4996-bd6b-78896b1026ed"). InnerVolumeSpecName "kube-api-access-mgk62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:57:38.643521 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.643485    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgk62\" (UniqueName: \"kubernetes.io/projected/6e6bd3b9-7218-4996-bd6b-78896b1026ed-kube-api-access-mgk62\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:38.643521 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.643516    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e6bd3b9-7218-4996-bd6b-78896b1026ed-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:38.643521 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:38.643525    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e6bd3b9-7218-4996-bd6b-78896b1026ed-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:39.136217 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:39.136182    2550 generic.go:356] "Generic (PLEG): container finished" podID="6e6bd3b9-7218-4996-bd6b-78896b1026ed" containerID="b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd" exitCode=0
Jul 20 19:57:39.136416 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:39.136225    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"
Jul 20 19:57:39.136416 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:39.136283    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p" event={"ID":"6e6bd3b9-7218-4996-bd6b-78896b1026ed","Type":"ContainerDied","Data":"b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd"}
Jul 20 19:57:39.136416 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:39.136337    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p" event={"ID":"6e6bd3b9-7218-4996-bd6b-78896b1026ed","Type":"ContainerDied","Data":"1f3e8f75bbe4e71688cc320906c49580b02c1f7b6b06a834b3dfb9b750ecf79f"}
Jul 20 19:57:39.136416 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:39.136364    2550 scope.go:117] "RemoveContainer" containerID="b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd"
Jul 20 19:57:39.144906 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:39.144886    2550 scope.go:117] "RemoveContainer" containerID="b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd"
Jul 20 19:57:39.145171 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:57:39.145151    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd\": container with ID starting with b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd not found: ID does not exist" containerID="b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd"
Jul 20 19:57:39.145225 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:39.145182    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd"} err="failed to get container status \"b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd\": rpc error: code = NotFound desc = could not find container \"b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd\": container with ID starting with b2f0a7f96fd1700e00e11933114a113eba8a52b379a153596fe141b3466156fd not found: ID does not exist"
Jul 20 19:57:39.157239 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:39.157219    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"]
Jul 20 19:57:39.160902 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:39.160883    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fd3c7e9a-56f594d75d-rjr4p"]
Jul 20 19:57:40.450070 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:40.450041    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e6bd3b9-7218-4996-bd6b-78896b1026ed" path="/var/lib/kubelet/pods/6e6bd3b9-7218-4996-bd6b-78896b1026ed/volumes"
Jul 20 19:57:49.914289 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:49.914233    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"]
Jul 20 19:57:49.914822 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:49.914472    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy" containerID="cri-o://5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7" gracePeriod=30
Jul 20 19:57:55.161046 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.161021    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:55.206045 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.205974    2550 generic.go:356] "Generic (PLEG): container finished" podID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerID="5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7" exitCode=0
Jul 20 19:57:55.206045 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.206015    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"
Jul 20 19:57:55.206211 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.206050    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" event={"ID":"8e3d3e16-ccdc-4e18-993f-ef60576eabdd","Type":"ContainerDied","Data":"5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7"}
Jul 20 19:57:55.206211 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.206102    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd" event={"ID":"8e3d3e16-ccdc-4e18-993f-ef60576eabdd","Type":"ContainerDied","Data":"2c6c06cc8e534c06de4dae7ea1503faced79d8daa000207e48354182ad36d07c"}
Jul 20 19:57:55.206211 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.206126    2550 scope.go:117] "RemoveContainer" containerID="5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7"
Jul 20 19:57:55.214373 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.214346    2550 scope.go:117] "RemoveContainer" containerID="5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018"
Jul 20 19:57:55.223897 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.223885    2550 scope.go:117] "RemoveContainer" containerID="5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7"
Jul 20 19:57:55.224333 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:57:55.224307    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7\": container with ID starting with 5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7 not found: ID does not exist" containerID="5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7"
Jul 20 19:57:55.224453 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.224343    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7"} err="failed to get container status \"5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7\": rpc error: code = NotFound desc = could not find container \"5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7\": container with ID starting with 5cad83a8f22270bcd5a77d0377b7e8a7e469c21354bf69c52ae99cc89aa384f7 not found: ID does not exist"
Jul 20 19:57:55.224453 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.224362    2550 scope.go:117] "RemoveContainer" containerID="5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018"
Jul 20 19:57:55.224620 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:57:55.224600    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018\": container with ID starting with 5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018 not found: ID does not exist" containerID="5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018"
Jul 20 19:57:55.224660 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.224627    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018"} err="failed to get container status \"5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018\": rpc error: code = NotFound desc = could not find container \"5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018\": container with ID starting with 5945cc3257764fb5e9c911b9f66102498f5c8646ae698e269594d1a19fe85018 not found: ID does not exist"
Jul 20 19:57:55.280970 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.280936    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-data\") pod \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") "
Jul 20 19:57:55.281143 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.280978    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-podinfo\") pod \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") "
Jul 20 19:57:55.281143 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281013    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-envoy\") pod \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") "
Jul 20 19:57:55.281143 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281031    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-certs\") pod \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") "
Jul 20 19:57:55.281143 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281063    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-token\") pod \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") "
Jul 20 19:57:55.281143 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281095    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istiod-ca-cert\") pod \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") "
Jul 20 19:57:55.281143 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281127    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-socket\") pod \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") "
Jul 20 19:57:55.281488 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281163    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-credential-socket\") pod \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") "
Jul 20 19:57:55.281488 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281199    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-549mx\" (UniqueName: \"kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-kube-api-access-549mx\") pod \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\" (UID: \"8e3d3e16-ccdc-4e18-993f-ef60576eabdd\") "
Jul 20 19:57:55.281488 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281403    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e3d3e16-ccdc-4e18-993f-ef60576eabdd" (UID: "8e3d3e16-ccdc-4e18-993f-ef60576eabdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:57:55.281637 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281531    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:55.281637 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281540    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e3d3e16-ccdc-4e18-993f-ef60576eabdd" (UID: "8e3d3e16-ccdc-4e18-993f-ef60576eabdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:57:55.281765 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281742    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e3d3e16-ccdc-4e18-993f-ef60576eabdd" (UID: "8e3d3e16-ccdc-4e18-993f-ef60576eabdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:57:55.281848 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281826    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e3d3e16-ccdc-4e18-993f-ef60576eabdd" (UID: "8e3d3e16-ccdc-4e18-993f-ef60576eabdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:57:55.281922 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.281897    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e3d3e16-ccdc-4e18-993f-ef60576eabdd" (UID: "8e3d3e16-ccdc-4e18-993f-ef60576eabdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:57:55.283344 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.283318    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e3d3e16-ccdc-4e18-993f-ef60576eabdd" (UID: "8e3d3e16-ccdc-4e18-993f-ef60576eabdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:57:55.283344 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.283330    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e3d3e16-ccdc-4e18-993f-ef60576eabdd" (UID: "8e3d3e16-ccdc-4e18-993f-ef60576eabdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 19:57:55.283567 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.283550    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e3d3e16-ccdc-4e18-993f-ef60576eabdd" (UID: "8e3d3e16-ccdc-4e18-993f-ef60576eabdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:57:55.283641 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.283626    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-kube-api-access-549mx" (OuterVolumeSpecName: "kube-api-access-549mx") pod "8e3d3e16-ccdc-4e18-993f-ef60576eabdd" (UID: "8e3d3e16-ccdc-4e18-993f-ef60576eabdd"). InnerVolumeSpecName "kube-api-access-549mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:57:55.382608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.382573    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-token\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:55.382608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.382600    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istiod-ca-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:55.382608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.382610    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-workload-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:55.382608 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.382619    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-credential-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:55.382864 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.382628    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-549mx\" (UniqueName: \"kubernetes.io/projected/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-kube-api-access-549mx\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:55.382864 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.382635    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-data\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:55.382864 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.382644    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-podinfo\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:55.382864 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.382651    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e3d3e16-ccdc-4e18-993f-ef60576eabdd-istio-envoy\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:57:55.529245 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.529213    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"]
Jul 20 19:57:55.538000 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:55.537970    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fd3c7e9a-openshift-default-765b684db7-f4lmd"]
Jul 20 19:57:56.450603 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:57:56.450568    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" path="/var/lib/kubelet/pods/8e3d3e16-ccdc-4e18-993f-ef60576eabdd/volumes"
Jul 20 19:58:10.094968 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.094934    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"]
Jul 20 19:58:10.095360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.095223    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy"
Jul 20 19:58:10.095360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.095233    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy"
Jul 20 19:58:10.095360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.095246    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy"
Jul 20 19:58:10.095360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.095251    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy"
Jul 20 19:58:10.095360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.095279    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e6bd3b9-7218-4996-bd6b-78896b1026ed" containerName="maas-api"
Jul 20 19:58:10.095360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.095285    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e6bd3b9-7218-4996-bd6b-78896b1026ed" containerName="maas-api"
Jul 20 19:58:10.095360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.095339    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy"
Jul 20 19:58:10.095360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.095348    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e6bd3b9-7218-4996-bd6b-78896b1026ed" containerName="maas-api"
Jul 20 19:58:10.095360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.095355    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e3d3e16-ccdc-4e18-993f-ef60576eabdd" containerName="istio-proxy"
Jul 20 19:58:10.098041 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.098018    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.101537 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.101516    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-723955e5-openshift-default-dockercfg-kzz67\""
Jul 20 19:58:10.101848 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.101834    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 20 19:58:10.118841 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.118807    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"]
Jul 20 19:58:10.205050 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.205011    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgkr\" (UniqueName: \"kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-kube-api-access-xqgkr\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.205215 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.205076    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-socket\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.205215 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.205096    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-podinfo\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.205215 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.205116    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-data\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.205215 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.205135    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-certs\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.205215 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.205151    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-credential-socket\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.205215 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.205168    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-token\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.205461 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.205278    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-envoy\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.205461 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.205317    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813c7905-7bcc-4e00-93b4-9a0c130419a4-istiod-ca-cert\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.305723 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.305681    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-socket\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.305908 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.305742    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-podinfo\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.305908 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.305766    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-data\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.305908 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.305787    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-certs\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.305908 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.305804    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-credential-socket\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.305908 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.305829    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-token\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.305908 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.305864    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-envoy\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.305908 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.305894    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813c7905-7bcc-4e00-93b4-9a0c130419a4-istiod-ca-cert\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.306412 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.305940    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgkr\" (UniqueName: \"kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-kube-api-access-xqgkr\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.306412 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.306103    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-socket\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.306412 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.306184    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-certs\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.306412 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.306314    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-credential-socket\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.306412 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.306347    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-data\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.306686 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.306668    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813c7905-7bcc-4e00-93b4-9a0c130419a4-istiod-ca-cert\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.308156 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.308132    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-envoy\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.308276 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.308213    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-podinfo\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.313430 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.313410    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-token\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.313634 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.313617    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgkr\" (UniqueName: \"kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-kube-api-access-xqgkr\") pod \"e2e-mt-723955e5-openshift-default-697856679-rqzgt\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") " pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.407559 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.407527    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:10.541187 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.541148    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"]
Jul 20 19:58:10.544197 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:58:10.544171    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813c7905_7bcc_4e00_93b4_9a0c130419a4.slice/crio-ce6eb49a8f63224869800d56d9747ba6e5c8eb591bac258cfe42a16f7ac574c5 WatchSource:0}: Error finding container ce6eb49a8f63224869800d56d9747ba6e5c8eb591bac258cfe42a16f7ac574c5: Status 404 returned error can't find the container with id ce6eb49a8f63224869800d56d9747ba6e5c8eb591bac258cfe42a16f7ac574c5
Jul 20 19:58:10.546421 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.546385    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:58:10.546530 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.546452    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:58:10.546530 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:10.546483    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:58:11.275685 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:11.275653    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" event={"ID":"813c7905-7bcc-4e00-93b4-9a0c130419a4","Type":"ContainerStarted","Data":"c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d"}
Jul 20 19:58:11.276071 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:11.275711    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" event={"ID":"813c7905-7bcc-4e00-93b4-9a0c130419a4","Type":"ContainerStarted","Data":"ce6eb49a8f63224869800d56d9747ba6e5c8eb591bac258cfe42a16f7ac574c5"}
Jul 20 19:58:11.302568 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:11.302523    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" podStartSLOduration=1.302506114 podStartE2EDuration="1.302506114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:58:11.301117541 +0000 UTC m=+2143.387068540" watchObservedRunningTime="2026-07-20 19:58:11.302506114 +0000 UTC m=+2143.388457112"
Jul 20 19:58:11.408715 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:11.408681    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:11.417622 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:11.417594    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:12.280640 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:12.280609    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:12.281661 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:12.281644    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:13.328966 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.328931    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"]
Jul 20 19:58:13.332817 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.332789    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:13.337286 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.337238    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-723955e5-serving-cert\""
Jul 20 19:58:13.343883 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.343862    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"]
Jul 20 19:58:13.432654 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.432614    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvbl\" (UniqueName: \"kubernetes.io/projected/70169514-6cfa-49e2-9f39-14c690ff20c5-kube-api-access-fhvbl\") pod \"maas-api-e2e-mt-723955e5-95c8498b9-5bbw9\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:13.432823 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.432683    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70169514-6cfa-49e2-9f39-14c690ff20c5-openshift-service-ca\") pod \"maas-api-e2e-mt-723955e5-95c8498b9-5bbw9\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:13.432823 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.432742    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70169514-6cfa-49e2-9f39-14c690ff20c5-maas-api-tls\") pod \"maas-api-e2e-mt-723955e5-95c8498b9-5bbw9\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:13.533943 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.533911    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70169514-6cfa-49e2-9f39-14c690ff20c5-openshift-service-ca\") pod \"maas-api-e2e-mt-723955e5-95c8498b9-5bbw9\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:13.534155 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.533978    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70169514-6cfa-49e2-9f39-14c690ff20c5-maas-api-tls\") pod \"maas-api-e2e-mt-723955e5-95c8498b9-5bbw9\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:13.534155 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.534054    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvbl\" (UniqueName: \"kubernetes.io/projected/70169514-6cfa-49e2-9f39-14c690ff20c5-kube-api-access-fhvbl\") pod \"maas-api-e2e-mt-723955e5-95c8498b9-5bbw9\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:13.534710 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.534675    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70169514-6cfa-49e2-9f39-14c690ff20c5-openshift-service-ca\") pod \"maas-api-e2e-mt-723955e5-95c8498b9-5bbw9\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:13.536661 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.536630    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70169514-6cfa-49e2-9f39-14c690ff20c5-maas-api-tls\") pod \"maas-api-e2e-mt-723955e5-95c8498b9-5bbw9\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:13.543847 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.543819    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvbl\" (UniqueName: \"kubernetes.io/projected/70169514-6cfa-49e2-9f39-14c690ff20c5-kube-api-access-fhvbl\") pod \"maas-api-e2e-mt-723955e5-95c8498b9-5bbw9\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:13.643763 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.643729    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:13.796092 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.796063    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"]
Jul 20 19:58:13.796727 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:58:13.796691    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70169514_6cfa_49e2_9f39_14c690ff20c5.slice/crio-8bea0358af4ed28406d9bdf88cf4246221d7a86f79369f1690079488d9e07133 WatchSource:0}: Error finding container 8bea0358af4ed28406d9bdf88cf4246221d7a86f79369f1690079488d9e07133: Status 404 returned error can't find the container with id 8bea0358af4ed28406d9bdf88cf4246221d7a86f79369f1690079488d9e07133
Jul 20 19:58:13.798221 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:13.798202    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 19:58:14.289682 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:14.289649    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9" event={"ID":"70169514-6cfa-49e2-9f39-14c690ff20c5","Type":"ContainerStarted","Data":"ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6"}
Jul 20 19:58:14.289840 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:14.289712    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9" event={"ID":"70169514-6cfa-49e2-9f39-14c690ff20c5","Type":"ContainerStarted","Data":"8bea0358af4ed28406d9bdf88cf4246221d7a86f79369f1690079488d9e07133"}
Jul 20 19:58:14.308378 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:14.308317    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9" podStartSLOduration=0.959766843 podStartE2EDuration="1.308295958s" totalImagesPullingTime="348.529115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:58:14.307001283 +0000 UTC m=+2146.392952307" watchObservedRunningTime="2026-07-20 19:58:14.308295958 +0000 UTC m=+2146.394246960"
Jul 20 19:58:15.293906 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:15.293874    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:21.303515 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:21.303433    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:58:27.802127 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:27.802040    2550 patch_prober.go:28] interesting pod/e2e-mt-723955e5-openshift-default-697856679-rqzgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:45004->10.133.0.48:15021: read: connection reset by peer" start-of-body=
Jul 20 19:58:27.802574 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:27.802119    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": read tcp 10.133.0.2:45004->10.133.0.48:15021: read: connection reset by peer"
Jul 20 19:58:28.386668 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:28.386636    2550 generic.go:356] "Generic (PLEG): container finished" podID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerID="c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d" exitCode=0
Jul 20 19:58:28.386668 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:28.386673    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" event={"ID":"813c7905-7bcc-4e00-93b4-9a0c130419a4","Type":"ContainerDied","Data":"c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d"}
Jul 20 19:58:28.387034 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:28.387017    2550 scope.go:117] "RemoveContainer" containerID="c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d"
Jul 20 19:58:28.387962 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:28.387931    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:58:28.388078 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:28.388026    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:58:28.388146 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:28.388081    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 19:58:29.393280 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:29.393232    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" event={"ID":"813c7905-7bcc-4e00-93b4-9a0c130419a4","Type":"ContainerStarted","Data":"b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea"}
Jul 20 19:58:29.408051 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:29.408019    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:29.408205 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:29.408187    2550 patch_prober.go:28] interesting pod/e2e-mt-723955e5-openshift-default-697856679-rqzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 20 19:58:29.408283 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:29.408230    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 20 19:58:30.408588 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:30.408552    2550 patch_prober.go:28] interesting pod/e2e-mt-723955e5-openshift-default-697856679-rqzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 20 19:58:30.408993 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:30.408624    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 20 19:58:31.408161 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:31.408125    2550 patch_prober.go:28] interesting pod/e2e-mt-723955e5-openshift-default-697856679-rqzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 20 19:58:31.408364 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:31.408179    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 20 19:58:32.408866 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:32.408825    2550 patch_prober.go:28] interesting pod/e2e-mt-723955e5-openshift-default-697856679-rqzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 20 19:58:32.409297 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:32.408877    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 20 19:58:33.408371 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:33.408328    2550 patch_prober.go:28] interesting pod/e2e-mt-723955e5-openshift-default-697856679-rqzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Jul 20 19:58:33.408560 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:33.408403    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Jul 20 19:58:34.411943 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.411910    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:34.412376 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.412302    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:34.413224 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.413203    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 19:58:34.427958 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.427926    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"]
Jul 20 19:58:34.431632 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.431606    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.436656 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.436631    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-723955e5\"/\"default-dockercfg-f5zhf\""
Jul 20 19:58:34.436965 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.436660    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-723955e5\"/\"kube-root-ca.crt\""
Jul 20 19:58:34.436965 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.436782    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-723955e5\"/\"openshift-service-ca.crt\""
Jul 20 19:58:34.437161 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.436857    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-723955e5\"/\"e2e-lifecycle-model-723955e5-kserve-self-signed-certs\""
Jul 20 19:58:34.441317 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.441296    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"]
Jul 20 19:58:34.503439 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.503405    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-tmp-dir\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.503439 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.503436    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727bec9f-3048-4aca-b678-8456fe38d7c9-tls-certs\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.503690 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.503484    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwfgh\" (UniqueName: \"kubernetes.io/projected/727bec9f-3048-4aca-b678-8456fe38d7c9-kube-api-access-rwfgh\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.503755 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.503697    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-model-cache\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.503840 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.503811    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-home\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.503934 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.503912    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-dshm\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.605246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.605210    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwfgh\" (UniqueName: \"kubernetes.io/projected/727bec9f-3048-4aca-b678-8456fe38d7c9-kube-api-access-rwfgh\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.605246 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.605276    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-model-cache\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.605547 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.605312    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-home\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.605547 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.605347    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-dshm\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.605547 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.605372    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-tmp-dir\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.605547 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.605400    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727bec9f-3048-4aca-b678-8456fe38d7c9-tls-certs\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.605795 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.605768    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-home\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.605867 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.605824    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-model-cache\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.605867 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.605848    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-tmp-dir\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.607652 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.607624    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-dshm\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.608078 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.608062    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727bec9f-3048-4aca-b678-8456fe38d7c9-tls-certs\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.613289 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.613247    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwfgh\" (UniqueName: \"kubernetes.io/projected/727bec9f-3048-4aca-b678-8456fe38d7c9-kube-api-access-rwfgh\") pod \"e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") " pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.745313 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.745211    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:34.881425 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:34.881397    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"]
Jul 20 19:58:34.883662 ip-10-0-138-227 kubenswrapper[2550]: W0720 19:58:34.883630    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod727bec9f_3048_4aca_b678_8456fe38d7c9.slice/crio-5a81a50907a88d91332aacacd0cbb8243801a91a49d45fe58195f8f3d6eacfd7 WatchSource:0}: Error finding container 5a81a50907a88d91332aacacd0cbb8243801a91a49d45fe58195f8f3d6eacfd7: Status 404 returned error can't find the container with id 5a81a50907a88d91332aacacd0cbb8243801a91a49d45fe58195f8f3d6eacfd7
Jul 20 19:58:35.422428 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:35.422383    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz" event={"ID":"727bec9f-3048-4aca-b678-8456fe38d7c9","Type":"ContainerStarted","Data":"42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6"}
Jul 20 19:58:35.422901 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:35.422436    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz" event={"ID":"727bec9f-3048-4aca-b678-8456fe38d7c9","Type":"ContainerStarted","Data":"5a81a50907a88d91332aacacd0cbb8243801a91a49d45fe58195f8f3d6eacfd7"}
Jul 20 19:58:35.441355 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:35.441303    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz" podStartSLOduration=1.441287248 podStartE2EDuration="1.441287248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 19:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 19:58:35.439759395 +0000 UTC m=+2167.525710383" watchObservedRunningTime="2026-07-20 19:58:35.441287248 +0000 UTC m=+2167.527238247"
Jul 20 19:58:44.745569 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:44.745527    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:44.745569 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:44.745578    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:44.758345 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:44.758314    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:58:45.486154 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:58:45.486126    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:59:11.849158 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:11.849119    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"]
Jul 20 19:59:11.849762 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:11.849430    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9" podUID="70169514-6cfa-49e2-9f39-14c690ff20c5" containerName="maas-api" containerID="cri-o://ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6" gracePeriod=30
Jul 20 19:59:12.092570 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.092543    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:59:12.140476 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.140449    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhvbl\" (UniqueName: \"kubernetes.io/projected/70169514-6cfa-49e2-9f39-14c690ff20c5-kube-api-access-fhvbl\") pod \"70169514-6cfa-49e2-9f39-14c690ff20c5\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") "
Jul 20 19:59:12.140617 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.140486    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70169514-6cfa-49e2-9f39-14c690ff20c5-openshift-service-ca\") pod \"70169514-6cfa-49e2-9f39-14c690ff20c5\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") "
Jul 20 19:59:12.140617 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.140543    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70169514-6cfa-49e2-9f39-14c690ff20c5-maas-api-tls\") pod \"70169514-6cfa-49e2-9f39-14c690ff20c5\" (UID: \"70169514-6cfa-49e2-9f39-14c690ff20c5\") "
Jul 20 19:59:12.140924 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.140900    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70169514-6cfa-49e2-9f39-14c690ff20c5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70169514-6cfa-49e2-9f39-14c690ff20c5" (UID: "70169514-6cfa-49e2-9f39-14c690ff20c5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 19:59:12.142559 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.142534    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70169514-6cfa-49e2-9f39-14c690ff20c5-kube-api-access-fhvbl" (OuterVolumeSpecName: "kube-api-access-fhvbl") pod "70169514-6cfa-49e2-9f39-14c690ff20c5" (UID: "70169514-6cfa-49e2-9f39-14c690ff20c5"). InnerVolumeSpecName "kube-api-access-fhvbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:59:12.142634 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.142579    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70169514-6cfa-49e2-9f39-14c690ff20c5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70169514-6cfa-49e2-9f39-14c690ff20c5" (UID: "70169514-6cfa-49e2-9f39-14c690ff20c5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:59:12.241160 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.241125    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvbl\" (UniqueName: \"kubernetes.io/projected/70169514-6cfa-49e2-9f39-14c690ff20c5-kube-api-access-fhvbl\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:59:12.241160 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.241154    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70169514-6cfa-49e2-9f39-14c690ff20c5-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:59:12.241160 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.241166    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70169514-6cfa-49e2-9f39-14c690ff20c5-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:59:12.593050 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.593012    2550 generic.go:356] "Generic (PLEG): container finished" podID="70169514-6cfa-49e2-9f39-14c690ff20c5" containerID="ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6" exitCode=0
Jul 20 19:59:12.593278 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.593061    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"
Jul 20 19:59:12.593278 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.593103    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9" event={"ID":"70169514-6cfa-49e2-9f39-14c690ff20c5","Type":"ContainerDied","Data":"ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6"}
Jul 20 19:59:12.593278 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.593156    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9" event={"ID":"70169514-6cfa-49e2-9f39-14c690ff20c5","Type":"ContainerDied","Data":"8bea0358af4ed28406d9bdf88cf4246221d7a86f79369f1690079488d9e07133"}
Jul 20 19:59:12.593278 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.593187    2550 scope.go:117] "RemoveContainer" containerID="ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6"
Jul 20 19:59:12.601390 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.601371    2550 scope.go:117] "RemoveContainer" containerID="ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6"
Jul 20 19:59:12.601689 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:59:12.601669    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6\": container with ID starting with ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6 not found: ID does not exist" containerID="ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6"
Jul 20 19:59:12.601745 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.601699    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6"} err="failed to get container status \"ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6\": rpc error: code = NotFound desc = could not find container \"ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6\": container with ID starting with ddc5bd44142325bf73c2f95b01844ca789a86f2387e6f99ca0cb2a79b6678ac6 not found: ID does not exist"
Jul 20 19:59:12.639635 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.639598    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"]
Jul 20 19:59:12.651360 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:12.651329    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-723955e5-95c8498b9-5bbw9"]
Jul 20 19:59:14.449913 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:14.449876    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70169514-6cfa-49e2-9f39-14c690ff20c5" path="/var/lib/kubelet/pods/70169514-6cfa-49e2-9f39-14c690ff20c5/volumes"
Jul 20 19:59:23.655700 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:23.655660    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"]
Jul 20 19:59:38.675899 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:38.675857    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz" podUID="727bec9f-3048-4aca-b678-8456fe38d7c9" containerName="main" containerID="cri-o://42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6" gracePeriod=45
Jul 20 19:59:38.923604 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:38.923578    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:59:39.077291 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077191    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-home\") pod \"727bec9f-3048-4aca-b678-8456fe38d7c9\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") "
Jul 20 19:59:39.077291 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077240    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727bec9f-3048-4aca-b678-8456fe38d7c9-tls-certs\") pod \"727bec9f-3048-4aca-b678-8456fe38d7c9\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") "
Jul 20 19:59:39.077291 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077290    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwfgh\" (UniqueName: \"kubernetes.io/projected/727bec9f-3048-4aca-b678-8456fe38d7c9-kube-api-access-rwfgh\") pod \"727bec9f-3048-4aca-b678-8456fe38d7c9\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") "
Jul 20 19:59:39.077584 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077340    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-model-cache\") pod \"727bec9f-3048-4aca-b678-8456fe38d7c9\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") "
Jul 20 19:59:39.077584 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077362    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-dshm\") pod \"727bec9f-3048-4aca-b678-8456fe38d7c9\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") "
Jul 20 19:59:39.077584 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077529    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-tmp-dir\") pod \"727bec9f-3048-4aca-b678-8456fe38d7c9\" (UID: \"727bec9f-3048-4aca-b678-8456fe38d7c9\") "
Jul 20 19:59:39.077584 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077546    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-home" (OuterVolumeSpecName: "home") pod "727bec9f-3048-4aca-b678-8456fe38d7c9" (UID: "727bec9f-3048-4aca-b678-8456fe38d7c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:59:39.077831 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077625    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "727bec9f-3048-4aca-b678-8456fe38d7c9" (UID: "727bec9f-3048-4aca-b678-8456fe38d7c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:59:39.077831 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077756    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "727bec9f-3048-4aca-b678-8456fe38d7c9" (UID: "727bec9f-3048-4aca-b678-8456fe38d7c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:59:39.077940 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077864    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-home\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:59:39.077940 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077877    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-model-cache\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:59:39.077940 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.077887    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-tmp-dir\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:59:39.079517 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.079495    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-dshm" (OuterVolumeSpecName: "dshm") pod "727bec9f-3048-4aca-b678-8456fe38d7c9" (UID: "727bec9f-3048-4aca-b678-8456fe38d7c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 19:59:39.079626 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.079549    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727bec9f-3048-4aca-b678-8456fe38d7c9-kube-api-access-rwfgh" (OuterVolumeSpecName: "kube-api-access-rwfgh") pod "727bec9f-3048-4aca-b678-8456fe38d7c9" (UID: "727bec9f-3048-4aca-b678-8456fe38d7c9"). InnerVolumeSpecName "kube-api-access-rwfgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 19:59:39.079670 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.079633    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/727bec9f-3048-4aca-b678-8456fe38d7c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "727bec9f-3048-4aca-b678-8456fe38d7c9" (UID: "727bec9f-3048-4aca-b678-8456fe38d7c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 19:59:39.179231 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.179191    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727bec9f-3048-4aca-b678-8456fe38d7c9-tls-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:59:39.179231 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.179225    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwfgh\" (UniqueName: \"kubernetes.io/projected/727bec9f-3048-4aca-b678-8456fe38d7c9-kube-api-access-rwfgh\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:59:39.179231 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.179237    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727bec9f-3048-4aca-b678-8456fe38d7c9-dshm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 19:59:39.713227 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.713193    2550 generic.go:356] "Generic (PLEG): container finished" podID="727bec9f-3048-4aca-b678-8456fe38d7c9" containerID="42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6" exitCode=0
Jul 20 19:59:39.713680 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.713235    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"
Jul 20 19:59:39.713680 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.713283    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz" event={"ID":"727bec9f-3048-4aca-b678-8456fe38d7c9","Type":"ContainerDied","Data":"42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6"}
Jul 20 19:59:39.713680 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.713327    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz" event={"ID":"727bec9f-3048-4aca-b678-8456fe38d7c9","Type":"ContainerDied","Data":"5a81a50907a88d91332aacacd0cbb8243801a91a49d45fe58195f8f3d6eacfd7"}
Jul 20 19:59:39.713680 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.713345    2550 scope.go:117] "RemoveContainer" containerID="42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6"
Jul 20 19:59:39.722625 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.722610    2550 scope.go:117] "RemoveContainer" containerID="42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6"
Jul 20 19:59:39.722877 ip-10-0-138-227 kubenswrapper[2550]: E0720 19:59:39.722857    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6\": container with ID starting with 42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6 not found: ID does not exist" containerID="42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6"
Jul 20 19:59:39.722927 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.722884    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6"} err="failed to get container status \"42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6\": rpc error: code = NotFound desc = could not find container \"42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6\": container with ID starting with 42e57d616bdd84e3897361bb4ec2cea6e8a7c4e1151deb4c35aab973a7356ed6 not found: ID does not exist"
Jul 20 19:59:39.735018 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.734996    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"]
Jul 20 19:59:39.739050 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:39.739030    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-723955e5/e2e-lifecycle-model-723955e5-kserve-9cfdff6bc-ss7vz"]
Jul 20 19:59:40.450410 ip-10-0-138-227 kubenswrapper[2550]: I0720 19:59:40.450375    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="727bec9f-3048-4aca-b678-8456fe38d7c9" path="/var/lib/kubelet/pods/727bec9f-3048-4aca-b678-8456fe38d7c9/volumes"
Jul 20 20:00:06.991560 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:06.991463    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"]
Jul 20 20:00:06.992072 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:06.991695    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy" containerID="cri-o://b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea" gracePeriod=30
Jul 20 20:00:12.243496 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.243465    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 20:00:12.366916 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.366823    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-credential-socket\") pod \"813c7905-7bcc-4e00-93b4-9a0c130419a4\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") "
Jul 20 20:00:12.366916 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.366880    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-socket\") pod \"813c7905-7bcc-4e00-93b4-9a0c130419a4\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") "
Jul 20 20:00:12.366916 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.366907    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-envoy\") pod \"813c7905-7bcc-4e00-93b4-9a0c130419a4\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") "
Jul 20 20:00:12.367244 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.366949    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-token\") pod \"813c7905-7bcc-4e00-93b4-9a0c130419a4\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") "
Jul 20 20:00:12.367244 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367005    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813c7905-7bcc-4e00-93b4-9a0c130419a4-istiod-ca-cert\") pod \"813c7905-7bcc-4e00-93b4-9a0c130419a4\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") "
Jul 20 20:00:12.367244 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367037    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-data\") pod \"813c7905-7bcc-4e00-93b4-9a0c130419a4\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") "
Jul 20 20:00:12.367244 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367079    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqgkr\" (UniqueName: \"kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-kube-api-access-xqgkr\") pod \"813c7905-7bcc-4e00-93b4-9a0c130419a4\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") "
Jul 20 20:00:12.367244 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367114    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-certs\") pod \"813c7905-7bcc-4e00-93b4-9a0c130419a4\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") "
Jul 20 20:00:12.367244 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367183    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-podinfo\") pod \"813c7905-7bcc-4e00-93b4-9a0c130419a4\" (UID: \"813c7905-7bcc-4e00-93b4-9a0c130419a4\") "
Jul 20 20:00:12.367244 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367180    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "813c7905-7bcc-4e00-93b4-9a0c130419a4" (UID: "813c7905-7bcc-4e00-93b4-9a0c130419a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:00:12.367244 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367230    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "813c7905-7bcc-4e00-93b4-9a0c130419a4" (UID: "813c7905-7bcc-4e00-93b4-9a0c130419a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:00:12.367677 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367400    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "813c7905-7bcc-4e00-93b4-9a0c130419a4" (UID: "813c7905-7bcc-4e00-93b4-9a0c130419a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:00:12.367677 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367450    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813c7905-7bcc-4e00-93b4-9a0c130419a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "813c7905-7bcc-4e00-93b4-9a0c130419a4" (UID: "813c7905-7bcc-4e00-93b4-9a0c130419a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 20:00:12.367677 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367524    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:00:12.367677 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367542    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-credential-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:00:12.367677 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.367557    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-workload-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:00:12.368115 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.368088    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "813c7905-7bcc-4e00-93b4-9a0c130419a4" (UID: "813c7905-7bcc-4e00-93b4-9a0c130419a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:00:12.369543 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.369514    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "813c7905-7bcc-4e00-93b4-9a0c130419a4" (UID: "813c7905-7bcc-4e00-93b4-9a0c130419a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 20:00:12.369798 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.369773    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-kube-api-access-xqgkr" (OuterVolumeSpecName: "kube-api-access-xqgkr") pod "813c7905-7bcc-4e00-93b4-9a0c130419a4" (UID: "813c7905-7bcc-4e00-93b4-9a0c130419a4"). InnerVolumeSpecName "kube-api-access-xqgkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:00:12.369902 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.369829    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "813c7905-7bcc-4e00-93b4-9a0c130419a4" (UID: "813c7905-7bcc-4e00-93b4-9a0c130419a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:00:12.369902 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.369833    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "813c7905-7bcc-4e00-93b4-9a0c130419a4" (UID: "813c7905-7bcc-4e00-93b4-9a0c130419a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:00:12.468244 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.468209    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-podinfo\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:00:12.468374 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.468250    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-envoy\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:00:12.468374 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.468282    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-token\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:00:12.468374 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.468295    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813c7905-7bcc-4e00-93b4-9a0c130419a4-istiod-ca-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:00:12.468374 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.468307    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813c7905-7bcc-4e00-93b4-9a0c130419a4-istio-data\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:00:12.468374 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.468321    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqgkr\" (UniqueName: \"kubernetes.io/projected/813c7905-7bcc-4e00-93b4-9a0c130419a4-kube-api-access-xqgkr\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:00:12.860160 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.860121    2550 generic.go:356] "Generic (PLEG): container finished" podID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerID="b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea" exitCode=0
Jul 20 20:00:12.860354 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.860192    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"
Jul 20 20:00:12.860354 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.860207    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" event={"ID":"813c7905-7bcc-4e00-93b4-9a0c130419a4","Type":"ContainerDied","Data":"b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea"}
Jul 20 20:00:12.860354 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.860282    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt" event={"ID":"813c7905-7bcc-4e00-93b4-9a0c130419a4","Type":"ContainerDied","Data":"ce6eb49a8f63224869800d56d9747ba6e5c8eb591bac258cfe42a16f7ac574c5"}
Jul 20 20:00:12.860354 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.860307    2550 scope.go:117] "RemoveContainer" containerID="b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea"
Jul 20 20:00:12.869249 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.869227    2550 scope.go:117] "RemoveContainer" containerID="c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d"
Jul 20 20:00:12.877787 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.877764    2550 scope.go:117] "RemoveContainer" containerID="b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea"
Jul 20 20:00:12.878097 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:00:12.878076    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea\": container with ID starting with b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea not found: ID does not exist" containerID="b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea"
Jul 20 20:00:12.878189 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.878110    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea"} err="failed to get container status \"b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea\": rpc error: code = NotFound desc = could not find container \"b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea\": container with ID starting with b12ff399a714159b032876af981e74178e61712b1b09150cc28cdf553d1c73ea not found: ID does not exist"
Jul 20 20:00:12.878189 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.878137    2550 scope.go:117] "RemoveContainer" containerID="c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d"
Jul 20 20:00:12.878461 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:00:12.878430    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d\": container with ID starting with c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d not found: ID does not exist" containerID="c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d"
Jul 20 20:00:12.878586 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.878468    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d"} err="failed to get container status \"c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d\": rpc error: code = NotFound desc = could not find container \"c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d\": container with ID starting with c30a342fa57187850ad287f0fdff39c7e9f467d153691873313c638266772e7d not found: ID does not exist"
Jul 20 20:00:12.880448 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.880426    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"]
Jul 20 20:00:12.885804 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:12.885778    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-723955e5-openshift-default-697856679-rqzgt"]
Jul 20 20:00:14.450885 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:00:14.450856    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" path="/var/lib/kubelet/pods/813c7905-7bcc-4e00-93b4-9a0c130419a4/volumes"
Jul 20 20:01:03.248018 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.247985    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"]
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248297    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="727bec9f-3048-4aca-b678-8456fe38d7c9" containerName="main"
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248309    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="727bec9f-3048-4aca-b678-8456fe38d7c9" containerName="main"
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248321    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy"
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248327    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy"
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248336    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70169514-6cfa-49e2-9f39-14c690ff20c5" containerName="maas-api"
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248341    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="70169514-6cfa-49e2-9f39-14c690ff20c5" containerName="maas-api"
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248355    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy"
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248360    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy"
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248404    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="70169514-6cfa-49e2-9f39-14c690ff20c5" containerName="maas-api"
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248413    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy"
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248418    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="727bec9f-3048-4aca-b678-8456fe38d7c9" containerName="main"
Jul 20 20:01:03.248717 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.248425    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="813c7905-7bcc-4e00-93b4-9a0c130419a4" containerName="istio-proxy"
Jul 20 20:01:03.251104 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.251085    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.253633 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.253614    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3e3435-openshift-default-dockercfg-mfk5n\""
Jul 20 20:01:03.254142 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.254126    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 20 20:01:03.280281 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.280230    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"]
Jul 20 20:01:03.283800 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.283768    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-token\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.283926 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.283821    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7defb2d-f60b-463e-bc9d-876c5531faaa-istiod-ca-cert\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.283926 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.283856    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-podinfo\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.283926 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.283910    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-socket\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.284051 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.283951    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-envoy\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.284051 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.283989    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-data\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.284115 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.284074    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-certs\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.284154 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.284109    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp2tq\" (UniqueName: \"kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-kube-api-access-qp2tq\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.284188 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.284165    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-credential-socket\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.384753 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.384714    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-token\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.384951 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.384756    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7defb2d-f60b-463e-bc9d-876c5531faaa-istiod-ca-cert\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.384951 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.384791    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-podinfo\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.384951 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.384815    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-socket\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.384951 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.384839    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-envoy\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.385152 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.385011    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-data\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.385322 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.385293    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-socket\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.385460 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.385349    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-certs\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.385460 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.385375    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp2tq\" (UniqueName: \"kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-kube-api-access-qp2tq\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.385460 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.385400    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-credential-socket\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.385710 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.385686    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-certs\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.385796 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.385724    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7defb2d-f60b-463e-bc9d-876c5531faaa-istiod-ca-cert\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.385796 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.385727    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-credential-socket\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.385796 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.385790    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-data\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.387228 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.387201    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-envoy\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.387452 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.387434    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-podinfo\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.398633 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.398610    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp2tq\" (UniqueName: \"kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-kube-api-access-qp2tq\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.403347 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.403325    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-token\") pod \"e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") " pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.561675 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.561583    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:03.707222 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.707194    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"]
Jul 20 20:01:03.709599 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:01:03.709571    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7defb2d_f60b_463e_bc9d_876c5531faaa.slice/crio-fb0033b7590c262e43b652dacffa4e90f7643e9d97bffcf8c00e4e1493d0b173 WatchSource:0}: Error finding container fb0033b7590c262e43b652dacffa4e90f7643e9d97bffcf8c00e4e1493d0b173: Status 404 returned error can't find the container with id fb0033b7590c262e43b652dacffa4e90f7643e9d97bffcf8c00e4e1493d0b173
Jul 20 20:01:03.711757 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.711727    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:01:03.711829 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.711809    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:01:03.711867 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:03.711836    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:01:04.085843 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:04.085798    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" event={"ID":"e7defb2d-f60b-463e-bc9d-876c5531faaa","Type":"ContainerStarted","Data":"796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f"}
Jul 20 20:01:04.086066 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:04.085856    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" event={"ID":"e7defb2d-f60b-463e-bc9d-876c5531faaa","Type":"ContainerStarted","Data":"fb0033b7590c262e43b652dacffa4e90f7643e9d97bffcf8c00e4e1493d0b173"}
Jul 20 20:01:04.112497 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:04.112326    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" podStartSLOduration=1.112304237 podStartE2EDuration="1.112304237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:01:04.10899329 +0000 UTC m=+2316.194944287" watchObservedRunningTime="2026-07-20 20:01:04.112304237 +0000 UTC m=+2316.198255243"
Jul 20 20:01:04.562346 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:04.562309    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:04.567228 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:04.567202    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:05.090954 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:05.090923    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:05.092032 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:05.092012    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:06.197094 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.197060    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"]
Jul 20 20:01:06.200181 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.200163    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:06.202872 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.202848    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3e3435-serving-cert\""
Jul 20 20:01:06.206098 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.206076    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4clg\" (UniqueName: \"kubernetes.io/projected/fe89626e-8290-414f-a056-2ea608abcf77-kube-api-access-v4clg\") pod \"maas-api-e2e-inf-a-3e3435-779b89b767-nq575\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:06.206186 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.206160    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe89626e-8290-414f-a056-2ea608abcf77-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3e3435-779b89b767-nq575\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:06.206230 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.206197    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe89626e-8290-414f-a056-2ea608abcf77-maas-api-tls\") pod \"maas-api-e2e-inf-a-3e3435-779b89b767-nq575\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:06.214210 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.214184    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"]
Jul 20 20:01:06.306572 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.306534    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe89626e-8290-414f-a056-2ea608abcf77-maas-api-tls\") pod \"maas-api-e2e-inf-a-3e3435-779b89b767-nq575\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:06.306787 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.306592    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4clg\" (UniqueName: \"kubernetes.io/projected/fe89626e-8290-414f-a056-2ea608abcf77-kube-api-access-v4clg\") pod \"maas-api-e2e-inf-a-3e3435-779b89b767-nq575\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:06.306787 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.306649    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe89626e-8290-414f-a056-2ea608abcf77-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3e3435-779b89b767-nq575\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:06.306787 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:01:06.306709    2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-serving-cert: secret "maas-api-e2e-inf-a-3e3435-serving-cert" not found
Jul 20 20:01:06.307008 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:01:06.306790    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe89626e-8290-414f-a056-2ea608abcf77-maas-api-tls podName:fe89626e-8290-414f-a056-2ea608abcf77 nodeName:}" failed. No retries permitted until 2026-07-20 20:01:06.806769024 +0000 UTC m=+2318.892720014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe89626e-8290-414f-a056-2ea608abcf77-maas-api-tls") pod "maas-api-e2e-inf-a-3e3435-779b89b767-nq575" (UID: "fe89626e-8290-414f-a056-2ea608abcf77") : secret "maas-api-e2e-inf-a-3e3435-serving-cert" not found
Jul 20 20:01:06.307366 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.307345    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe89626e-8290-414f-a056-2ea608abcf77-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3e3435-779b89b767-nq575\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:06.317413 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.317384    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4clg\" (UniqueName: \"kubernetes.io/projected/fe89626e-8290-414f-a056-2ea608abcf77-kube-api-access-v4clg\") pod \"maas-api-e2e-inf-a-3e3435-779b89b767-nq575\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:06.810816 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.810782    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe89626e-8290-414f-a056-2ea608abcf77-maas-api-tls\") pod \"maas-api-e2e-inf-a-3e3435-779b89b767-nq575\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:06.813427 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:06.813398    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe89626e-8290-414f-a056-2ea608abcf77-maas-api-tls\") pod \"maas-api-e2e-inf-a-3e3435-779b89b767-nq575\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:07.110344 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:07.110240    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:07.246346 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:07.246312    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"]
Jul 20 20:01:07.248850 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:01:07.248814    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe89626e_8290_414f_a056_2ea608abcf77.slice/crio-46520b9f43fb1adac6c9ead59f0726f7bcb1d421942bd357b35a72e151e0da44 WatchSource:0}: Error finding container 46520b9f43fb1adac6c9ead59f0726f7bcb1d421942bd357b35a72e151e0da44: Status 404 returned error can't find the container with id 46520b9f43fb1adac6c9ead59f0726f7bcb1d421942bd357b35a72e151e0da44
Jul 20 20:01:08.104858 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:08.104824    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575" event={"ID":"fe89626e-8290-414f-a056-2ea608abcf77","Type":"ContainerStarted","Data":"550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2"}
Jul 20 20:01:08.105020 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:08.104880    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575" event={"ID":"fe89626e-8290-414f-a056-2ea608abcf77","Type":"ContainerStarted","Data":"46520b9f43fb1adac6c9ead59f0726f7bcb1d421942bd357b35a72e151e0da44"}
Jul 20 20:01:08.105020 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:08.104940    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:08.127746 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:08.127692    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575" podStartSLOduration=1.425457562 podStartE2EDuration="2.127671631s" totalImagesPullingTime="702.214069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:01:08.126297578 +0000 UTC m=+2320.212248575" watchObservedRunningTime="2026-07-20 20:01:08.127671631 +0000 UTC m=+2320.213622607"
Jul 20 20:01:14.043893 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.043849    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"]
Jul 20 20:01:14.048195 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.047831    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:14.050401 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.050377    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-dc9964-serving-cert\""
Jul 20 20:01:14.063490 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.063458    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"]
Jul 20 20:01:14.073075 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.073046    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eddf34-b637-4d9b-b3c7-e81ca3681989-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:14.073203 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.073163    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16eddf34-b637-4d9b-b3c7-e81ca3681989-maas-api-tls\") pod \"maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:14.073203 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.073191    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d56t\" (UniqueName: \"kubernetes.io/projected/16eddf34-b637-4d9b-b3c7-e81ca3681989-kube-api-access-2d56t\") pod \"maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:14.114342 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.114311    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:01:14.174455 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.174413    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eddf34-b637-4d9b-b3c7-e81ca3681989-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:14.174640 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.174593    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16eddf34-b637-4d9b-b3c7-e81ca3681989-maas-api-tls\") pod \"maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:14.174640 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.174636    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d56t\" (UniqueName: \"kubernetes.io/projected/16eddf34-b637-4d9b-b3c7-e81ca3681989-kube-api-access-2d56t\") pod \"maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:14.174771 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:01:14.174689    2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-serving-cert: secret "maas-api-e2e-inf-b-dc9964-serving-cert" not found
Jul 20 20:01:14.174771 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:01:14.174757    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16eddf34-b637-4d9b-b3c7-e81ca3681989-maas-api-tls podName:16eddf34-b637-4d9b-b3c7-e81ca3681989 nodeName:}" failed. No retries permitted until 2026-07-20 20:01:14.674735347 +0000 UTC m=+2326.760686323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16eddf34-b637-4d9b-b3c7-e81ca3681989-maas-api-tls") pod "maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9" (UID: "16eddf34-b637-4d9b-b3c7-e81ca3681989") : secret "maas-api-e2e-inf-b-dc9964-serving-cert" not found
Jul 20 20:01:14.175133 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.175116    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eddf34-b637-4d9b-b3c7-e81ca3681989-openshift-service-ca\") pod \"maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:14.185821 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.185795    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d56t\" (UniqueName: \"kubernetes.io/projected/16eddf34-b637-4d9b-b3c7-e81ca3681989-kube-api-access-2d56t\") pod \"maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:14.686649 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.686612    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16eddf34-b637-4d9b-b3c7-e81ca3681989-maas-api-tls\") pod \"maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:14.689156 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.689132    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16eddf34-b637-4d9b-b3c7-e81ca3681989-maas-api-tls\") pod \"maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:14.959915 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:14.959815    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:15.113396 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:15.113371    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"]
Jul 20 20:01:15.128458 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:01:15.127526    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16eddf34_b637_4d9b_b3c7_e81ca3681989.slice/crio-b6059cecd129a22ac5a4620be012cf88d997faf578711546e94fc18344ea93e9 WatchSource:0}: Error finding container b6059cecd129a22ac5a4620be012cf88d997faf578711546e94fc18344ea93e9: Status 404 returned error can't find the container with id b6059cecd129a22ac5a4620be012cf88d997faf578711546e94fc18344ea93e9
Jul 20 20:01:15.149383 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:15.149335    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9" event={"ID":"16eddf34-b637-4d9b-b3c7-e81ca3681989","Type":"ContainerStarted","Data":"b6059cecd129a22ac5a4620be012cf88d997faf578711546e94fc18344ea93e9"}
Jul 20 20:01:16.155622 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:16.155588    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9" event={"ID":"16eddf34-b637-4d9b-b3c7-e81ca3681989","Type":"ContainerStarted","Data":"f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea"}
Jul 20 20:01:16.156024 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:16.155663    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:16.173274 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:16.173207    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9" podStartSLOduration=1.5651795 podStartE2EDuration="2.173187366s" totalImagesPullingTime="608.007866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:01:16.171943441 +0000 UTC m=+2328.257894474" watchObservedRunningTime="2026-07-20 20:01:16.173187366 +0000 UTC m=+2328.259138364"
Jul 20 20:01:20.092052 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:20.091951    2550 patch_prober.go:28] interesting pod/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 20 20:01:20.092052 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:20.092003    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 20 20:01:20.176110 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:20.176072    2550 generic.go:356] "Generic (PLEG): container finished" podID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerID="796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f" exitCode=0
Jul 20 20:01:20.176288 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:20.176144    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" event={"ID":"e7defb2d-f60b-463e-bc9d-876c5531faaa","Type":"ContainerDied","Data":"796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f"}
Jul 20 20:01:20.176542 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:20.176528    2550 scope.go:117] "RemoveContainer" containerID="796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f"
Jul 20 20:01:20.177618 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:20.177589    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:01:20.177690 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:20.177667    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:01:20.177729 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:20.177704    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:01:21.182199 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.182166    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" event={"ID":"e7defb2d-f60b-463e-bc9d-876c5531faaa","Type":"ContainerStarted","Data":"7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88"}
Jul 20 20:01:21.562175 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.562080    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:21.562512 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.562487    2550 patch_prober.go:28] interesting pod/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 20 20:01:21.562617 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.562545    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 20 20:01:21.834134 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.834029    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"]
Jul 20 20:01:21.836966 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.836942    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:21.842807 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.842771    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3e3435\"/\"openshift-service-ca.crt\""
Jul 20 20:01:21.842931 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.842873    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3e3435\"/\"kube-root-ca.crt\""
Jul 20 20:01:21.843469 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.843441    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3e3435\"/\"default-dockercfg-tk2x8\""
Jul 20 20:01:21.843748 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.843729    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3e3435\"/\"test-model-3e3435-kserve-self-signed-certs\""
Jul 20 20:01:21.848274 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.847677    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"]
Jul 20 20:01:21.951735 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.951699    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-model-cache\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:21.951735 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.951737    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tmp-dir\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:21.951951 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.951752    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-dshm\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:21.951951 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.951845    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tls-certs\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:21.951951 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.951875    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-home\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:21.951951 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:21.951903    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpg9\" (UniqueName: \"kubernetes.io/projected/77cbe208-a213-4e49-a6a7-f93c62abd2ed-kube-api-access-5xpg9\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.053097 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.053055    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tls-certs\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.053097 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.053097    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-home\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.053365 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.053122    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpg9\" (UniqueName: \"kubernetes.io/projected/77cbe208-a213-4e49-a6a7-f93c62abd2ed-kube-api-access-5xpg9\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.053365 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.053178    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-model-cache\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.053365 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.053192    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tmp-dir\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.053365 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.053207    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-dshm\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.053674 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.053639    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-home\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.053728 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.053680    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tmp-dir\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.053769 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.053754    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-model-cache\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.055509 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.055484    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-dshm\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.055769 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.055751    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tls-certs\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.062066 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.062046    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpg9\" (UniqueName: \"kubernetes.io/projected/77cbe208-a213-4e49-a6a7-f93c62abd2ed-kube-api-access-5xpg9\") pod \"test-model-3e3435-kserve-5fcd6975bc-2lknn\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") " pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.153849 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.153810    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:22.166373 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.166346    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:01:22.335281 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.334409    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"]
Jul 20 20:01:22.345244 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:01:22.345210    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77cbe208_a213_4e49_a6a7_f93c62abd2ed.slice/crio-229a8296ed5c7115ab77e119c48079a03ec5ef7986a44d9ba365e090eb006b41 WatchSource:0}: Error finding container 229a8296ed5c7115ab77e119c48079a03ec5ef7986a44d9ba365e090eb006b41: Status 404 returned error can't find the container with id 229a8296ed5c7115ab77e119c48079a03ec5ef7986a44d9ba365e090eb006b41
Jul 20 20:01:22.562321 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.562196    2550 patch_prober.go:28] interesting pod/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 20 20:01:22.562321 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:22.562279    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 20 20:01:23.193318 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:23.193277    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn" event={"ID":"77cbe208-a213-4e49-a6a7-f93c62abd2ed","Type":"ContainerStarted","Data":"889073801f208d1882e65b04315131909be58d078decf79684cfa5bf094d9c2a"}
Jul 20 20:01:23.193497 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:23.193328    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn" event={"ID":"77cbe208-a213-4e49-a6a7-f93c62abd2ed","Type":"ContainerStarted","Data":"229a8296ed5c7115ab77e119c48079a03ec5ef7986a44d9ba365e090eb006b41"}
Jul 20 20:01:23.212433 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:23.212372    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn" podStartSLOduration=2.212352197 podStartE2EDuration="2.212352197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:01:23.21126613 +0000 UTC m=+2335.297217121" watchObservedRunningTime="2026-07-20 20:01:23.212352197 +0000 UTC m=+2335.298303197"
Jul 20 20:01:23.562648 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:23.562555    2550 patch_prober.go:28] interesting pod/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 20 20:01:23.562648 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:23.562611    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 20 20:01:24.562027 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:24.561982    2550 patch_prober.go:28] interesting pod/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 20 20:01:24.562210 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:24.562061    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 20 20:01:25.566597 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:25.566564    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:25.566982 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:25.566831    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:25.567554 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:25.567533    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:01:32.154373 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:32.154334    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:32.154868 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:32.154679    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:32.168668 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:32.168646    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:32.246711 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:32.246682    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:01:57.606496 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.606454    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"]
Jul 20 20:01:57.609675 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.609649    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.612780 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.612754    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dc9964\"/\"default-dockercfg-zfjr7\""
Jul 20 20:01:57.612780 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.612773    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dc9964\"/\"openshift-service-ca.crt\""
Jul 20 20:01:57.613507 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.613489    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dc9964\"/\"test-model-dc9964-kserve-self-signed-certs\""
Jul 20 20:01:57.613591 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.613533    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dc9964\"/\"kube-root-ca.crt\""
Jul 20 20:01:57.624204 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.624182    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"]
Jul 20 20:01:57.661966 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.661932    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-tmp-dir\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.661966 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.661969    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-home\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.662221 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.662011    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-dshm\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.662221 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.662100    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mssdv\" (UniqueName: \"kubernetes.io/projected/dff1322d-951a-425f-82a1-6bc12fb98652-kube-api-access-mssdv\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.662221 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.662142    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-model-cache\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.662221 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.662198    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff1322d-951a-425f-82a1-6bc12fb98652-tls-certs\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.762787 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.762752    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mssdv\" (UniqueName: \"kubernetes.io/projected/dff1322d-951a-425f-82a1-6bc12fb98652-kube-api-access-mssdv\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.762960 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.762794    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-model-cache\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.762960 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.762912    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff1322d-951a-425f-82a1-6bc12fb98652-tls-certs\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.763098 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.762958    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-tmp-dir\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.763098 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.762986    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-home\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.763098 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.763046    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-dshm\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.763229 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.763208    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-model-cache\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.763370 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.763346    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-home\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.763510 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.763494    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-tmp-dir\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.765193 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.765175    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-dshm\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.765303 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.765229    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff1322d-951a-425f-82a1-6bc12fb98652-tls-certs\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.771740 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.771721    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mssdv\" (UniqueName: \"kubernetes.io/projected/dff1322d-951a-425f-82a1-6bc12fb98652-kube-api-access-mssdv\") pod \"test-model-dc9964-kserve-5c6d4c4bc5-2mv7s\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") " pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:57.918361 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:57.918322    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:01:58.063843 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:58.063809    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"]
Jul 20 20:01:58.067385 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:01:58.067351    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddff1322d_951a_425f_82a1_6bc12fb98652.slice/crio-d26194de054c8ca017616cb17637a7e7bf87e91529951d2cb46a8d48ab917d6d WatchSource:0}: Error finding container d26194de054c8ca017616cb17637a7e7bf87e91529951d2cb46a8d48ab917d6d: Status 404 returned error can't find the container with id d26194de054c8ca017616cb17637a7e7bf87e91529951d2cb46a8d48ab917d6d
Jul 20 20:01:58.351656 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:58.351564    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s" event={"ID":"dff1322d-951a-425f-82a1-6bc12fb98652","Type":"ContainerStarted","Data":"55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb"}
Jul 20 20:01:58.351656 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:58.351607    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s" event={"ID":"dff1322d-951a-425f-82a1-6bc12fb98652","Type":"ContainerStarted","Data":"d26194de054c8ca017616cb17637a7e7bf87e91529951d2cb46a8d48ab917d6d"}
Jul 20 20:01:58.373626 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:01:58.373572    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s" podStartSLOduration=1.373557267 podStartE2EDuration="1.373557267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:01:58.372250963 +0000 UTC m=+2370.458201962" watchObservedRunningTime="2026-07-20 20:01:58.373557267 +0000 UTC m=+2370.459508264"
Jul 20 20:02:07.919444 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:07.919405    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:02:07.919444 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:07.919450    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:02:07.932024 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:07.932001    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:02:08.406529 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:08.406500    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:02:28.495180 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:28.495149    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 20:02:28.500794 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:28.500770    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 20:02:38.326049 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:38.326011    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"]
Jul 20 20:02:39.234320 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:39.234281    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"]
Jul 20 20:02:49.894112 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:49.894037    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"]
Jul 20 20:02:49.894527 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:49.894271    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575" podUID="fe89626e-8290-414f-a056-2ea608abcf77" containerName="maas-api" containerID="cri-o://550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2" gracePeriod=30
Jul 20 20:02:50.138033 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.138009    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:02:50.228541 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.228462    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe89626e-8290-414f-a056-2ea608abcf77-openshift-service-ca\") pod \"fe89626e-8290-414f-a056-2ea608abcf77\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") "
Jul 20 20:02:50.228541 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.228531    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe89626e-8290-414f-a056-2ea608abcf77-maas-api-tls\") pod \"fe89626e-8290-414f-a056-2ea608abcf77\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") "
Jul 20 20:02:50.228773 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.228573    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4clg\" (UniqueName: \"kubernetes.io/projected/fe89626e-8290-414f-a056-2ea608abcf77-kube-api-access-v4clg\") pod \"fe89626e-8290-414f-a056-2ea608abcf77\" (UID: \"fe89626e-8290-414f-a056-2ea608abcf77\") "
Jul 20 20:02:50.228833 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.228791    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe89626e-8290-414f-a056-2ea608abcf77-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe89626e-8290-414f-a056-2ea608abcf77" (UID: "fe89626e-8290-414f-a056-2ea608abcf77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 20:02:50.230614 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.230586    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe89626e-8290-414f-a056-2ea608abcf77-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe89626e-8290-414f-a056-2ea608abcf77" (UID: "fe89626e-8290-414f-a056-2ea608abcf77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:02:50.230825 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.230808    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe89626e-8290-414f-a056-2ea608abcf77-kube-api-access-v4clg" (OuterVolumeSpecName: "kube-api-access-v4clg") pod "fe89626e-8290-414f-a056-2ea608abcf77" (UID: "fe89626e-8290-414f-a056-2ea608abcf77"). InnerVolumeSpecName "kube-api-access-v4clg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:02:50.329352 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.329319    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe89626e-8290-414f-a056-2ea608abcf77-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:50.329352 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.329347    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4clg\" (UniqueName: \"kubernetes.io/projected/fe89626e-8290-414f-a056-2ea608abcf77-kube-api-access-v4clg\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:50.329352 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.329357    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe89626e-8290-414f-a056-2ea608abcf77-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:50.573312 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.573204    2550 generic.go:356] "Generic (PLEG): container finished" podID="fe89626e-8290-414f-a056-2ea608abcf77" containerID="550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2" exitCode=0
Jul 20 20:02:50.573312 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.573245    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"
Jul 20 20:02:50.573312 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.573300    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575" event={"ID":"fe89626e-8290-414f-a056-2ea608abcf77","Type":"ContainerDied","Data":"550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2"}
Jul 20 20:02:50.573638 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.573351    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575" event={"ID":"fe89626e-8290-414f-a056-2ea608abcf77","Type":"ContainerDied","Data":"46520b9f43fb1adac6c9ead59f0726f7bcb1d421942bd357b35a72e151e0da44"}
Jul 20 20:02:50.573638 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.573372    2550 scope.go:117] "RemoveContainer" containerID="550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2"
Jul 20 20:02:50.583509 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.583476    2550 scope.go:117] "RemoveContainer" containerID="550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2"
Jul 20 20:02:50.583901 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:02:50.583881    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2\": container with ID starting with 550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2 not found: ID does not exist" containerID="550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2"
Jul 20 20:02:50.583958 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.583909    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2"} err="failed to get container status \"550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2\": rpc error: code = NotFound desc = could not find container \"550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2\": container with ID starting with 550d0934838b3af18bdc894a5e0213dda8bc1477da22cacb7b99c0de7b91f8e2 not found: ID does not exist"
Jul 20 20:02:50.589154 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.589121    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"]
Jul 20 20:02:50.593066 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:50.593042    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3e3435-779b89b767-nq575"]
Jul 20 20:02:52.451675 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:52.451644    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe89626e-8290-414f-a056-2ea608abcf77" path="/var/lib/kubelet/pods/fe89626e-8290-414f-a056-2ea608abcf77/volumes"
Jul 20 20:02:53.345178 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.345134    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn" podUID="77cbe208-a213-4e49-a6a7-f93c62abd2ed" containerName="main" containerID="cri-o://889073801f208d1882e65b04315131909be58d078decf79684cfa5bf094d9c2a" gracePeriod=45
Jul 20 20:02:53.590482 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.590455    2550 generic.go:356] "Generic (PLEG): container finished" podID="77cbe208-a213-4e49-a6a7-f93c62abd2ed" containerID="889073801f208d1882e65b04315131909be58d078decf79684cfa5bf094d9c2a" exitCode=0
Jul 20 20:02:53.590823 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.590530    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn" event={"ID":"77cbe208-a213-4e49-a6a7-f93c62abd2ed","Type":"ContainerDied","Data":"889073801f208d1882e65b04315131909be58d078decf79684cfa5bf094d9c2a"}
Jul 20 20:02:53.590823 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.590579    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn" event={"ID":"77cbe208-a213-4e49-a6a7-f93c62abd2ed","Type":"ContainerDied","Data":"229a8296ed5c7115ab77e119c48079a03ec5ef7986a44d9ba365e090eb006b41"}
Jul 20 20:02:53.590823 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.590599    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="229a8296ed5c7115ab77e119c48079a03ec5ef7986a44d9ba365e090eb006b41"
Jul 20 20:02:53.590823 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.590726    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:02:53.657270 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.657223    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-home\") pod \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") "
Jul 20 20:02:53.657434 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.657276    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tmp-dir\") pod \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") "
Jul 20 20:02:53.657434 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.657303    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tls-certs\") pod \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") "
Jul 20 20:02:53.657434 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.657322    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xpg9\" (UniqueName: \"kubernetes.io/projected/77cbe208-a213-4e49-a6a7-f93c62abd2ed-kube-api-access-5xpg9\") pod \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") "
Jul 20 20:02:53.657434 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.657394    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-model-cache\") pod \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") "
Jul 20 20:02:53.657645 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.657457    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-dshm\") pod \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\" (UID: \"77cbe208-a213-4e49-a6a7-f93c62abd2ed\") "
Jul 20 20:02:53.657645 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.657486    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-home" (OuterVolumeSpecName: "home") pod "77cbe208-a213-4e49-a6a7-f93c62abd2ed" (UID: "77cbe208-a213-4e49-a6a7-f93c62abd2ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:02:53.657645 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.657596    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77cbe208-a213-4e49-a6a7-f93c62abd2ed" (UID: "77cbe208-a213-4e49-a6a7-f93c62abd2ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:02:53.657793 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.657701    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "77cbe208-a213-4e49-a6a7-f93c62abd2ed" (UID: "77cbe208-a213-4e49-a6a7-f93c62abd2ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:02:53.657793 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.657756    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-home\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:53.657793 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.657779    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tmp-dir\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:53.659598 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.659570    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-dshm" (OuterVolumeSpecName: "dshm") pod "77cbe208-a213-4e49-a6a7-f93c62abd2ed" (UID: "77cbe208-a213-4e49-a6a7-f93c62abd2ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:02:53.659667 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.659649    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77cbe208-a213-4e49-a6a7-f93c62abd2ed" (UID: "77cbe208-a213-4e49-a6a7-f93c62abd2ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:02:53.659708 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.659694    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cbe208-a213-4e49-a6a7-f93c62abd2ed-kube-api-access-5xpg9" (OuterVolumeSpecName: "kube-api-access-5xpg9") pod "77cbe208-a213-4e49-a6a7-f93c62abd2ed" (UID: "77cbe208-a213-4e49-a6a7-f93c62abd2ed"). InnerVolumeSpecName "kube-api-access-5xpg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:02:53.758623 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.758586    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77cbe208-a213-4e49-a6a7-f93c62abd2ed-tls-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:53.758623 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.758615    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xpg9\" (UniqueName: \"kubernetes.io/projected/77cbe208-a213-4e49-a6a7-f93c62abd2ed-kube-api-access-5xpg9\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:53.758623 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.758627    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-model-cache\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:53.758623 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:53.758636    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77cbe208-a213-4e49-a6a7-f93c62abd2ed-dshm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:54.253586 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.253511    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s" podUID="dff1322d-951a-425f-82a1-6bc12fb98652" containerName="main" containerID="cri-o://55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb" gracePeriod=45
Jul 20 20:02:54.500769 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.500745    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:02:54.567882 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.567764    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-tmp-dir\") pod \"dff1322d-951a-425f-82a1-6bc12fb98652\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") "
Jul 20 20:02:54.567882 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.567813    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-dshm\") pod \"dff1322d-951a-425f-82a1-6bc12fb98652\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") "
Jul 20 20:02:54.567882 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.567856    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-home\") pod \"dff1322d-951a-425f-82a1-6bc12fb98652\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") "
Jul 20 20:02:54.567882 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.567883    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-model-cache\") pod \"dff1322d-951a-425f-82a1-6bc12fb98652\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") "
Jul 20 20:02:54.568332 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.567934    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mssdv\" (UniqueName: \"kubernetes.io/projected/dff1322d-951a-425f-82a1-6bc12fb98652-kube-api-access-mssdv\") pod \"dff1322d-951a-425f-82a1-6bc12fb98652\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") "
Jul 20 20:02:54.568332 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.567982    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff1322d-951a-425f-82a1-6bc12fb98652-tls-certs\") pod \"dff1322d-951a-425f-82a1-6bc12fb98652\" (UID: \"dff1322d-951a-425f-82a1-6bc12fb98652\") "
Jul 20 20:02:54.568332 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.568099    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dff1322d-951a-425f-82a1-6bc12fb98652" (UID: "dff1322d-951a-425f-82a1-6bc12fb98652"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:02:54.568332 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.568237    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-home" (OuterVolumeSpecName: "home") pod "dff1322d-951a-425f-82a1-6bc12fb98652" (UID: "dff1322d-951a-425f-82a1-6bc12fb98652"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:02:54.568526 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.568336    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-model-cache" (OuterVolumeSpecName: "model-cache") pod "dff1322d-951a-425f-82a1-6bc12fb98652" (UID: "dff1322d-951a-425f-82a1-6bc12fb98652"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:02:54.568604 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.568584    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-tmp-dir\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:54.568659 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.568612    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-home\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:54.568659 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.568629    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-model-cache\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:54.570538 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.570502    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-dshm" (OuterVolumeSpecName: "dshm") pod "dff1322d-951a-425f-82a1-6bc12fb98652" (UID: "dff1322d-951a-425f-82a1-6bc12fb98652"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:02:54.570538 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.570517    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff1322d-951a-425f-82a1-6bc12fb98652-kube-api-access-mssdv" (OuterVolumeSpecName: "kube-api-access-mssdv") pod "dff1322d-951a-425f-82a1-6bc12fb98652" (UID: "dff1322d-951a-425f-82a1-6bc12fb98652"). InnerVolumeSpecName "kube-api-access-mssdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:02:54.570721 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.570545    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff1322d-951a-425f-82a1-6bc12fb98652-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dff1322d-951a-425f-82a1-6bc12fb98652" (UID: "dff1322d-951a-425f-82a1-6bc12fb98652"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:02:54.595697 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.595665    2550 generic.go:356] "Generic (PLEG): container finished" podID="dff1322d-951a-425f-82a1-6bc12fb98652" containerID="55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb" exitCode=0
Jul 20 20:02:54.596057 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.595724    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"
Jul 20 20:02:54.596057 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.595749    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s" event={"ID":"dff1322d-951a-425f-82a1-6bc12fb98652","Type":"ContainerDied","Data":"55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb"}
Jul 20 20:02:54.596057 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.595789    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s" event={"ID":"dff1322d-951a-425f-82a1-6bc12fb98652","Type":"ContainerDied","Data":"d26194de054c8ca017616cb17637a7e7bf87e91529951d2cb46a8d48ab917d6d"}
Jul 20 20:02:54.596057 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.595805    2550 scope.go:117] "RemoveContainer" containerID="55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb"
Jul 20 20:02:54.596057 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.595938    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"
Jul 20 20:02:54.604313 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.604298    2550 scope.go:117] "RemoveContainer" containerID="55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb"
Jul 20 20:02:54.604544 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:02:54.604527    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb\": container with ID starting with 55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb not found: ID does not exist" containerID="55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb"
Jul 20 20:02:54.604594 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.604553    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb"} err="failed to get container status \"55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb\": rpc error: code = NotFound desc = could not find container \"55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb\": container with ID starting with 55d530e2e65be6eeedbc24433275f8683395dbed5dd596322181cb405e9708eb not found: ID does not exist"
Jul 20 20:02:54.613002 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.612977    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"]
Jul 20 20:02:54.618818 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.618798    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3e3435/test-model-3e3435-kserve-5fcd6975bc-2lknn"]
Jul 20 20:02:54.632049 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.632027    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"]
Jul 20 20:02:54.635115 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.635096    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-dc9964/test-model-dc9964-kserve-5c6d4c4bc5-2mv7s"]
Jul 20 20:02:54.669530 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.669504    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dff1322d-951a-425f-82a1-6bc12fb98652-dshm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:54.669530 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.669527    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mssdv\" (UniqueName: \"kubernetes.io/projected/dff1322d-951a-425f-82a1-6bc12fb98652-kube-api-access-mssdv\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:54.669666 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:54.669537    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dff1322d-951a-425f-82a1-6bc12fb98652-tls-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:02:56.450124 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:56.450091    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77cbe208-a213-4e49-a6a7-f93c62abd2ed" path="/var/lib/kubelet/pods/77cbe208-a213-4e49-a6a7-f93c62abd2ed/volumes"
Jul 20 20:02:56.450519 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:02:56.450472    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff1322d-951a-425f-82a1-6bc12fb98652" path="/var/lib/kubelet/pods/dff1322d-951a-425f-82a1-6bc12fb98652/volumes"
Jul 20 20:03:01.911338 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:01.911306    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"]
Jul 20 20:03:01.911809 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:01.911502    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy" containerID="cri-o://7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88" gracePeriod=30
Jul 20 20:03:07.158402 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.158378    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:03:07.278301 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278185    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-socket\") pod \"e7defb2d-f60b-463e-bc9d-876c5531faaa\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") "
Jul 20 20:03:07.278301 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278225    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-podinfo\") pod \"e7defb2d-f60b-463e-bc9d-876c5531faaa\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") "
Jul 20 20:03:07.278301 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278281    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-credential-socket\") pod \"e7defb2d-f60b-463e-bc9d-876c5531faaa\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") "
Jul 20 20:03:07.278301 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278303    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7defb2d-f60b-463e-bc9d-876c5531faaa-istiod-ca-cert\") pod \"e7defb2d-f60b-463e-bc9d-876c5531faaa\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") "
Jul 20 20:03:07.278651 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278326    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp2tq\" (UniqueName: \"kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-kube-api-access-qp2tq\") pod \"e7defb2d-f60b-463e-bc9d-876c5531faaa\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") "
Jul 20 20:03:07.278651 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278370    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-token\") pod \"e7defb2d-f60b-463e-bc9d-876c5531faaa\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") "
Jul 20 20:03:07.278651 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278393    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-envoy\") pod \"e7defb2d-f60b-463e-bc9d-876c5531faaa\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") "
Jul 20 20:03:07.278651 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278415    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-data\") pod \"e7defb2d-f60b-463e-bc9d-876c5531faaa\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") "
Jul 20 20:03:07.278651 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278468    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-certs\") pod \"e7defb2d-f60b-463e-bc9d-876c5531faaa\" (UID: \"e7defb2d-f60b-463e-bc9d-876c5531faaa\") "
Jul 20 20:03:07.278651 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278539    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e7defb2d-f60b-463e-bc9d-876c5531faaa" (UID: "e7defb2d-f60b-463e-bc9d-876c5531faaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:03:07.278651 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278592    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e7defb2d-f60b-463e-bc9d-876c5531faaa" (UID: "e7defb2d-f60b-463e-bc9d-876c5531faaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:03:07.279009 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278717    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e7defb2d-f60b-463e-bc9d-876c5531faaa" (UID: "e7defb2d-f60b-463e-bc9d-876c5531faaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:03:07.279009 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278741    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-credential-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:07.279009 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278762    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:07.279009 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.278792    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7defb2d-f60b-463e-bc9d-876c5531faaa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e7defb2d-f60b-463e-bc9d-876c5531faaa" (UID: "e7defb2d-f60b-463e-bc9d-876c5531faaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 20:03:07.279620 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.279595    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-data" (OuterVolumeSpecName: "istio-data") pod "e7defb2d-f60b-463e-bc9d-876c5531faaa" (UID: "e7defb2d-f60b-463e-bc9d-876c5531faaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:03:07.280554 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.280529    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e7defb2d-f60b-463e-bc9d-876c5531faaa" (UID: "e7defb2d-f60b-463e-bc9d-876c5531faaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 20:03:07.280630 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.280553    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e7defb2d-f60b-463e-bc9d-876c5531faaa" (UID: "e7defb2d-f60b-463e-bc9d-876c5531faaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:03:07.280847 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.280814    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-kube-api-access-qp2tq" (OuterVolumeSpecName: "kube-api-access-qp2tq") pod "e7defb2d-f60b-463e-bc9d-876c5531faaa" (UID: "e7defb2d-f60b-463e-bc9d-876c5531faaa"). InnerVolumeSpecName "kube-api-access-qp2tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:03:07.280943 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.280848    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-token" (OuterVolumeSpecName: "istio-token") pod "e7defb2d-f60b-463e-bc9d-876c5531faaa" (UID: "e7defb2d-f60b-463e-bc9d-876c5531faaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:03:07.379573 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.379543    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-token\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:07.379573 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.379569    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-envoy\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:07.379573 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.379578    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-data\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:07.379792 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.379586    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7defb2d-f60b-463e-bc9d-876c5531faaa-workload-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:07.379792 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.379595    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7defb2d-f60b-463e-bc9d-876c5531faaa-istio-podinfo\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:07.379792 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.379602    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7defb2d-f60b-463e-bc9d-876c5531faaa-istiod-ca-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:07.379792 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.379611    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp2tq\" (UniqueName: \"kubernetes.io/projected/e7defb2d-f60b-463e-bc9d-876c5531faaa-kube-api-access-qp2tq\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:07.654389 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.654358    2550 generic.go:356] "Generic (PLEG): container finished" podID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerID="7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88" exitCode=0
Jul 20 20:03:07.654522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.654405    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"
Jul 20 20:03:07.654522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.654426    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" event={"ID":"e7defb2d-f60b-463e-bc9d-876c5531faaa","Type":"ContainerDied","Data":"7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88"}
Jul 20 20:03:07.654522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.654467    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx" event={"ID":"e7defb2d-f60b-463e-bc9d-876c5531faaa","Type":"ContainerDied","Data":"fb0033b7590c262e43b652dacffa4e90f7643e9d97bffcf8c00e4e1493d0b173"}
Jul 20 20:03:07.654522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.654486    2550 scope.go:117] "RemoveContainer" containerID="7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88"
Jul 20 20:03:07.662853 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.662838    2550 scope.go:117] "RemoveContainer" containerID="796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f"
Jul 20 20:03:07.670133 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.670117    2550 scope.go:117] "RemoveContainer" containerID="7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88"
Jul 20 20:03:07.670368 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:03:07.670347    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88\": container with ID starting with 7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88 not found: ID does not exist" containerID="7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88"
Jul 20 20:03:07.670429 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.670375    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88"} err="failed to get container status \"7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88\": rpc error: code = NotFound desc = could not find container \"7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88\": container with ID starting with 7ae1a2639d28733952e567341755a8ee19925c5effb2b1046182d133fd31ec88 not found: ID does not exist"
Jul 20 20:03:07.670429 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.670393    2550 scope.go:117] "RemoveContainer" containerID="796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f"
Jul 20 20:03:07.670606 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:03:07.670589    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f\": container with ID starting with 796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f not found: ID does not exist" containerID="796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f"
Jul 20 20:03:07.670644 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.670610    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f"} err="failed to get container status \"796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f\": rpc error: code = NotFound desc = could not find container \"796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f\": container with ID starting with 796fd4633d6890e6ccf587afe4f6d48f987975531fbf71772ba4d3df8d03528f not found: ID does not exist"
Jul 20 20:03:07.679000 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.678975    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"]
Jul 20 20:03:07.683501 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:07.683480    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3e3435-openshift-default-684d76fd67-zd6dx"]
Jul 20 20:03:08.454299 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:08.454266    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" path="/var/lib/kubelet/pods/e7defb2d-f60b-463e-bc9d-876c5531faaa/volumes"
Jul 20 20:03:12.869366 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:12.869322    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"]
Jul 20 20:03:12.869935 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:12.869579    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9" podUID="16eddf34-b637-4d9b-b3c7-e81ca3681989" containerName="maas-api" containerID="cri-o://f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea" gracePeriod=30
Jul 20 20:03:13.115860 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.115835    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:03:13.228119 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.228083    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16eddf34-b637-4d9b-b3c7-e81ca3681989-maas-api-tls\") pod \"16eddf34-b637-4d9b-b3c7-e81ca3681989\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") "
Jul 20 20:03:13.228306 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.228169    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d56t\" (UniqueName: \"kubernetes.io/projected/16eddf34-b637-4d9b-b3c7-e81ca3681989-kube-api-access-2d56t\") pod \"16eddf34-b637-4d9b-b3c7-e81ca3681989\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") "
Jul 20 20:03:13.228306 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.228203    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eddf34-b637-4d9b-b3c7-e81ca3681989-openshift-service-ca\") pod \"16eddf34-b637-4d9b-b3c7-e81ca3681989\" (UID: \"16eddf34-b637-4d9b-b3c7-e81ca3681989\") "
Jul 20 20:03:13.228603 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.228574    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16eddf34-b637-4d9b-b3c7-e81ca3681989-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16eddf34-b637-4d9b-b3c7-e81ca3681989" (UID: "16eddf34-b637-4d9b-b3c7-e81ca3681989"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 20:03:13.230165 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.230143    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16eddf34-b637-4d9b-b3c7-e81ca3681989-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16eddf34-b637-4d9b-b3c7-e81ca3681989" (UID: "16eddf34-b637-4d9b-b3c7-e81ca3681989"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:03:13.230252 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.230197    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16eddf34-b637-4d9b-b3c7-e81ca3681989-kube-api-access-2d56t" (OuterVolumeSpecName: "kube-api-access-2d56t") pod "16eddf34-b637-4d9b-b3c7-e81ca3681989" (UID: "16eddf34-b637-4d9b-b3c7-e81ca3681989"). InnerVolumeSpecName "kube-api-access-2d56t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:03:13.328824 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.328791    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16eddf34-b637-4d9b-b3c7-e81ca3681989-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:13.328824 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.328818    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16eddf34-b637-4d9b-b3c7-e81ca3681989-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:13.328824 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.328831    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d56t\" (UniqueName: \"kubernetes.io/projected/16eddf34-b637-4d9b-b3c7-e81ca3681989-kube-api-access-2d56t\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:03:13.686911 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.686874    2550 generic.go:356] "Generic (PLEG): container finished" podID="16eddf34-b637-4d9b-b3c7-e81ca3681989" containerID="f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea" exitCode=0
Jul 20 20:03:13.687113 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.686919    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"
Jul 20 20:03:13.687113 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.686928    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9" event={"ID":"16eddf34-b637-4d9b-b3c7-e81ca3681989","Type":"ContainerDied","Data":"f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea"}
Jul 20 20:03:13.687113 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.686969    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9" event={"ID":"16eddf34-b637-4d9b-b3c7-e81ca3681989","Type":"ContainerDied","Data":"b6059cecd129a22ac5a4620be012cf88d997faf578711546e94fc18344ea93e9"}
Jul 20 20:03:13.687113 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.686992    2550 scope.go:117] "RemoveContainer" containerID="f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea"
Jul 20 20:03:13.696010 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.695990    2550 scope.go:117] "RemoveContainer" containerID="f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea"
Jul 20 20:03:13.696270 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:03:13.696239    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea\": container with ID starting with f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea not found: ID does not exist" containerID="f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea"
Jul 20 20:03:13.696355 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.696325    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea"} err="failed to get container status \"f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea\": rpc error: code = NotFound desc = could not find container \"f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea\": container with ID starting with f895d51c24875bb9e1fbbb9a5901219613b9eef7e9c9c94462e46ac6df49fbea not found: ID does not exist"
Jul 20 20:03:13.707938 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.707907    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"]
Jul 20 20:03:13.711854 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:13.711832    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-dc9964-68fcf94cf8-hldn9"]
Jul 20 20:03:14.450712 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:14.450669    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16eddf34-b637-4d9b-b3c7-e81ca3681989" path="/var/lib/kubelet/pods/16eddf34-b637-4d9b-b3c7-e81ca3681989/volumes"
Jul 20 20:03:25.063379 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063344    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"]
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063643    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy"
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063655    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy"
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063663    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe89626e-8290-414f-a056-2ea608abcf77" containerName="maas-api"
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063668    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe89626e-8290-414f-a056-2ea608abcf77" containerName="maas-api"
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063678    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77cbe208-a213-4e49-a6a7-f93c62abd2ed" containerName="main"
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063685    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="77cbe208-a213-4e49-a6a7-f93c62abd2ed" containerName="main"
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063696    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16eddf34-b637-4d9b-b3c7-e81ca3681989" containerName="maas-api"
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063701    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="16eddf34-b637-4d9b-b3c7-e81ca3681989" containerName="maas-api"
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063713    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff1322d-951a-425f-82a1-6bc12fb98652" containerName="main"
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063718    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff1322d-951a-425f-82a1-6bc12fb98652" containerName="main"
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063724    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy"
Jul 20 20:03:25.063774 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063729    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy"
Jul 20 20:03:25.064162 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063792    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy"
Jul 20 20:03:25.064162 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063800    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="16eddf34-b637-4d9b-b3c7-e81ca3681989" containerName="maas-api"
Jul 20 20:03:25.064162 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063805    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="77cbe208-a213-4e49-a6a7-f93c62abd2ed" containerName="main"
Jul 20 20:03:25.064162 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063812    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff1322d-951a-425f-82a1-6bc12fb98652" containerName="main"
Jul 20 20:03:25.064162 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.063820    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe89626e-8290-414f-a056-2ea608abcf77" containerName="maas-api"
Jul 20 20:03:25.067898 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.067880    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.070125 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.070101    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 20 20:03:25.070267 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.070146    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-da8568-openshift-default-dockercfg-svnfr\""
Jul 20 20:03:25.079208 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.078921    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"]
Jul 20 20:03:25.225988 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.225948    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-token\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.225988 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.225992    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-credential-socket\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.226201 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.226013    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-podinfo\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.226201 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.226036    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-certs\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.226201 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.226121    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-data\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.226201 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.226161    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istiod-ca-cert\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.226201 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.226179    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-envoy\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.226398 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.226206    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-socket\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.226398 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.226227    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2bn\" (UniqueName: \"kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-kube-api-access-wm2bn\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327081 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.326988    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-data\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327081 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327036    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istiod-ca-cert\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327081 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327062    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-envoy\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327431 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327093    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-socket\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327431 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327119    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2bn\" (UniqueName: \"kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-kube-api-access-wm2bn\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327431 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327168    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-token\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327431 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327196    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-credential-socket\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327431 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327223    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-podinfo\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327431 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327282    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-certs\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327790 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327446    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-data\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327790 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327459    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-socket\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327790 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327576    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-credential-socket\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.327964 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327810    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istiod-ca-cert\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.328032 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.327991    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-certs\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.329361 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.329346    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-envoy\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.329697 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.329681    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-podinfo\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.335184 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.335158    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-token\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.335300 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.335247    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2bn\" (UniqueName: \"kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-kube-api-access-wm2bn\") pod \"e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") " pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.380896 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.380857    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:25.514704 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.514634    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"]
Jul 20 20:03:25.518414 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:03:25.518374    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17b32f8_2a78_4a20_8817_4bfbc83fcf38.slice/crio-e8747ddb9afc75390198622d2bd938b255f51cca319d189c373b5dbc1b48f4fd WatchSource:0}: Error finding container e8747ddb9afc75390198622d2bd938b255f51cca319d189c373b5dbc1b48f4fd: Status 404 returned error can't find the container with id e8747ddb9afc75390198622d2bd938b255f51cca319d189c373b5dbc1b48f4fd
Jul 20 20:03:25.520462 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.520429    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:25.520531 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.520499    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:25.520531 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.520527    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:25.741533 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.741491    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" event={"ID":"a17b32f8-2a78-4a20-8817-4bfbc83fcf38","Type":"ContainerStarted","Data":"b851d3d3497829181fd2dbdf3203306a003996a92c96120d7e424ae145db6880"}
Jul 20 20:03:25.741722 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.741559    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" event={"ID":"a17b32f8-2a78-4a20-8817-4bfbc83fcf38","Type":"ContainerStarted","Data":"e8747ddb9afc75390198622d2bd938b255f51cca319d189c373b5dbc1b48f4fd"}
Jul 20 20:03:25.764802 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:25.764747    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" podStartSLOduration=0.764727662 podStartE2EDuration="764.727662ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:03:25.763235165 +0000 UTC m=+2457.849186166" watchObservedRunningTime="2026-07-20 20:03:25.764727662 +0000 UTC m=+2457.850678660"
Jul 20 20:03:26.381214 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:26.381176    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:26.386003 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:26.385976    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:26.746835 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:26.746740    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:26.747638 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:26.747619    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:27.877666 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:27.877636    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"]
Jul 20 20:03:27.878181 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:27.878161    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7defb2d-f60b-463e-bc9d-876c5531faaa" containerName="istio-proxy"
Jul 20 20:03:27.881106 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:27.881085    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:27.883644 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:27.883622    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-da8568-serving-cert\""
Jul 20 20:03:27.890378 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:27.890356    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"]
Jul 20 20:03:28.052432 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.052392    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d09052-f768-4e65-a420-e7b25c7bbe7e-openshift-service-ca\") pod \"maas-api-e2e-api-a-da8568-9f5969759-g4h4g\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:28.052636 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.052447    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d09052-f768-4e65-a420-e7b25c7bbe7e-maas-api-tls\") pod \"maas-api-e2e-api-a-da8568-9f5969759-g4h4g\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:28.052636 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.052521    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46qq\" (UniqueName: \"kubernetes.io/projected/59d09052-f768-4e65-a420-e7b25c7bbe7e-kube-api-access-x46qq\") pod \"maas-api-e2e-api-a-da8568-9f5969759-g4h4g\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:28.153509 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.153409    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46qq\" (UniqueName: \"kubernetes.io/projected/59d09052-f768-4e65-a420-e7b25c7bbe7e-kube-api-access-x46qq\") pod \"maas-api-e2e-api-a-da8568-9f5969759-g4h4g\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:28.153509 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.153498    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d09052-f768-4e65-a420-e7b25c7bbe7e-openshift-service-ca\") pod \"maas-api-e2e-api-a-da8568-9f5969759-g4h4g\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:28.153781 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.153556    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d09052-f768-4e65-a420-e7b25c7bbe7e-maas-api-tls\") pod \"maas-api-e2e-api-a-da8568-9f5969759-g4h4g\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:28.153781 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:03:28.153670    2550 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-serving-cert: secret "maas-api-e2e-api-a-da8568-serving-cert" not found
Jul 20 20:03:28.153781 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:03:28.153740    2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59d09052-f768-4e65-a420-e7b25c7bbe7e-maas-api-tls podName:59d09052-f768-4e65-a420-e7b25c7bbe7e nodeName:}" failed. No retries permitted until 2026-07-20 20:03:28.653715856 +0000 UTC m=+2460.739666833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59d09052-f768-4e65-a420-e7b25c7bbe7e-maas-api-tls") pod "maas-api-e2e-api-a-da8568-9f5969759-g4h4g" (UID: "59d09052-f768-4e65-a420-e7b25c7bbe7e") : secret "maas-api-e2e-api-a-da8568-serving-cert" not found
Jul 20 20:03:28.154149 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.154128    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d09052-f768-4e65-a420-e7b25c7bbe7e-openshift-service-ca\") pod \"maas-api-e2e-api-a-da8568-9f5969759-g4h4g\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:28.163877 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.163845    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46qq\" (UniqueName: \"kubernetes.io/projected/59d09052-f768-4e65-a420-e7b25c7bbe7e-kube-api-access-x46qq\") pod \"maas-api-e2e-api-a-da8568-9f5969759-g4h4g\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:28.657896 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.657874    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d09052-f768-4e65-a420-e7b25c7bbe7e-maas-api-tls\") pod \"maas-api-e2e-api-a-da8568-9f5969759-g4h4g\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:28.660516 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.660494    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d09052-f768-4e65-a420-e7b25c7bbe7e-maas-api-tls\") pod \"maas-api-e2e-api-a-da8568-9f5969759-g4h4g\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:28.792736 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.792705    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:28.930763 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.930684    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"]
Jul 20 20:03:28.934485 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:03:28.934448    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d09052_f768_4e65_a420_e7b25c7bbe7e.slice/crio-b22634178bcc410ee30d84068869796637ee11a9a29b821d452043919d824408 WatchSource:0}: Error finding container b22634178bcc410ee30d84068869796637ee11a9a29b821d452043919d824408: Status 404 returned error can't find the container with id b22634178bcc410ee30d84068869796637ee11a9a29b821d452043919d824408
Jul 20 20:03:28.936003 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:28.935985    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 20:03:29.761009 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:29.760972    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g" event={"ID":"59d09052-f768-4e65-a420-e7b25c7bbe7e","Type":"ContainerStarted","Data":"460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115"}
Jul 20 20:03:29.761230 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:29.761037    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g" event={"ID":"59d09052-f768-4e65-a420-e7b25c7bbe7e","Type":"ContainerStarted","Data":"b22634178bcc410ee30d84068869796637ee11a9a29b821d452043919d824408"}
Jul 20 20:03:29.761230 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:29.761064    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:29.779781 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:29.779734    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g" podStartSLOduration=2.176254173 podStartE2EDuration="2.779719915s" totalImagesPullingTime="603.465742ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:03:29.778233148 +0000 UTC m=+2461.864184145" watchObservedRunningTime="2026-07-20 20:03:29.779719915 +0000 UTC m=+2461.865670912"
Jul 20 20:03:32.947092 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.947058    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"]
Jul 20 20:03:32.950430 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.950414    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:32.953137 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.953111    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-75e28c-openshift-default-dockercfg-9dxlj\""
Jul 20 20:03:32.965616 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.965586    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"]
Jul 20 20:03:32.992081 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.992047    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-token\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:32.992081 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.992083    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istiod-ca-cert\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:32.992315 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.992118    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-credential-socket\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:32.992315 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.992155    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-data\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:32.992315 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.992177    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-socket\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:32.992315 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.992211    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-podinfo\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:32.992315 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.992229    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-envoy\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:32.992315 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.992285    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdzrd\" (UniqueName: \"kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-kube-api-access-bdzrd\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:32.992315 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:32.992313    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-certs\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.092748 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.092710    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-socket\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.092748 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.092754    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-podinfo\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.092968 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.092869    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-envoy\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.092968 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.092914    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdzrd\" (UniqueName: \"kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-kube-api-access-bdzrd\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.092968 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.092958    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-certs\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.093093 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.093003    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-token\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.093093 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.093035    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istiod-ca-cert\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.093093 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.093074    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-credential-socket\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.093305 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.093126    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-socket\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.093305 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.093129    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-data\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.093513 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.093490    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-credential-socket\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.093568 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.093540    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-certs\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.093612 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.093568    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-data\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.094009 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.093985    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istiod-ca-cert\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.095426 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.095404    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-envoy\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.095656 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.095638    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-podinfo\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.100218 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.100197    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-token\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.100633 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.100613    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdzrd\" (UniqueName: \"kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-kube-api-access-bdzrd\") pod \"e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") " pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.260696 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.260600    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:33.395189 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.395160    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"]
Jul 20 20:03:33.395990 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:03:33.395965    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd2af0c_6b2c_4f0d_a008_dba2da7ff6ab.slice/crio-b77075c1e69dd5310da2ab1ac9a7769d7cfdd64dd26f854084459b70bd1a37d4 WatchSource:0}: Error finding container b77075c1e69dd5310da2ab1ac9a7769d7cfdd64dd26f854084459b70bd1a37d4: Status 404 returned error can't find the container with id b77075c1e69dd5310da2ab1ac9a7769d7cfdd64dd26f854084459b70bd1a37d4
Jul 20 20:03:33.398577 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.398545    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:33.398646 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.398611    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:33.398646 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.398638    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:33.780996 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.780959    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" event={"ID":"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab","Type":"ContainerStarted","Data":"ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535"}
Jul 20 20:03:33.781185 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.781018    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" event={"ID":"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab","Type":"ContainerStarted","Data":"b77075c1e69dd5310da2ab1ac9a7769d7cfdd64dd26f854084459b70bd1a37d4"}
Jul 20 20:03:33.828785 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:33.828742    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" podStartSLOduration=1.828726385 podStartE2EDuration="1.828726385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:03:33.828214286 +0000 UTC m=+2465.914165286" watchObservedRunningTime="2026-07-20 20:03:33.828726385 +0000 UTC m=+2465.914677382"
Jul 20 20:03:34.261720 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:34.261682    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:34.266551 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:34.266524    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:34.785830 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:34.785800    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:34.786913 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:34.786894    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:35.770321 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:35.770286    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:03:35.975644 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:35.975606    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"]
Jul 20 20:03:35.979317 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:35.979293    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:35.981383 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:35.981357    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-75e28c-serving-cert\""
Jul 20 20:03:35.990455 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:35.990430    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"]
Jul 20 20:03:36.015681 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.015644    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-openshift-service-ca\") pod \"maas-api-e2e-api-b-75e28c-78c94c746d-49nck\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:36.015681 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.015687    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5g8n\" (UniqueName: \"kubernetes.io/projected/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-kube-api-access-w5g8n\") pod \"maas-api-e2e-api-b-75e28c-78c94c746d-49nck\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:36.015894 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.015773    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-maas-api-tls\") pod \"maas-api-e2e-api-b-75e28c-78c94c746d-49nck\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:36.116810 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.116704    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-maas-api-tls\") pod \"maas-api-e2e-api-b-75e28c-78c94c746d-49nck\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:36.116810 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.116782    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-openshift-service-ca\") pod \"maas-api-e2e-api-b-75e28c-78c94c746d-49nck\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:36.117049 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.116819    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5g8n\" (UniqueName: \"kubernetes.io/projected/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-kube-api-access-w5g8n\") pod \"maas-api-e2e-api-b-75e28c-78c94c746d-49nck\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:36.117553 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.117527    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-openshift-service-ca\") pod \"maas-api-e2e-api-b-75e28c-78c94c746d-49nck\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:36.119267 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.119243    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-maas-api-tls\") pod \"maas-api-e2e-api-b-75e28c-78c94c746d-49nck\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:36.126910 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.126882    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5g8n\" (UniqueName: \"kubernetes.io/projected/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-kube-api-access-w5g8n\") pod \"maas-api-e2e-api-b-75e28c-78c94c746d-49nck\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:36.293758 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.293714    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:36.435824 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.435796    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"]
Jul 20 20:03:36.438329 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:03:36.438295    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8dd5535_70d8_43c9_a13d_2d8ee0ad157f.slice/crio-8c81184a32d41e353cbe37320bccb931c59e42a6eb96708233fb767adefe36b1 WatchSource:0}: Error finding container 8c81184a32d41e353cbe37320bccb931c59e42a6eb96708233fb767adefe36b1: Status 404 returned error can't find the container with id 8c81184a32d41e353cbe37320bccb931c59e42a6eb96708233fb767adefe36b1
Jul 20 20:03:36.797075 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:36.797037    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck" event={"ID":"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f","Type":"ContainerStarted","Data":"8c81184a32d41e353cbe37320bccb931c59e42a6eb96708233fb767adefe36b1"}
Jul 20 20:03:37.802755 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:37.802710    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck" event={"ID":"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f","Type":"ContainerStarted","Data":"bc283fbc0f41f38debb0186a56beb4e37503db86ff4f9894b8f5a92b8ada84c8"}
Jul 20 20:03:37.803195 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:37.802791    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:37.824770 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:37.824671    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck" podStartSLOduration=2.506735623 podStartE2EDuration="2.824634352s" totalImagesPullingTime="317.898729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:03:37.822708475 +0000 UTC m=+2469.908659473" watchObservedRunningTime="2026-07-20 20:03:37.824634352 +0000 UTC m=+2469.910585350"
Jul 20 20:03:42.788461 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:42.788416    2550 patch_prober.go:28] interesting pod/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 20 20:03:42.788886 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:42.788478    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 20:03:43.812503 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:43.812413    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:03:43.833245 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:43.833206    2550 generic.go:356] "Generic (PLEG): container finished" podID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerID="b851d3d3497829181fd2dbdf3203306a003996a92c96120d7e424ae145db6880" exitCode=0
Jul 20 20:03:43.833245 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:43.833245    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" event={"ID":"a17b32f8-2a78-4a20-8817-4bfbc83fcf38","Type":"ContainerDied","Data":"b851d3d3497829181fd2dbdf3203306a003996a92c96120d7e424ae145db6880"}
Jul 20 20:03:43.833664 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:43.833646    2550 scope.go:117] "RemoveContainer" containerID="b851d3d3497829181fd2dbdf3203306a003996a92c96120d7e424ae145db6880"
Jul 20 20:03:43.834688 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:43.834659    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:43.834776 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:43.834735    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:43.834776 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:43.834767    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:44.842534 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:44.842493    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" event={"ID":"a17b32f8-2a78-4a20-8817-4bfbc83fcf38","Type":"ContainerStarted","Data":"a76dbbde1606cdeaa8b1f246588f60f73c3a4eabde6c7a0b7776184c1f2567f4"}
Jul 20 20:03:45.382081 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:45.382036    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:45.382352 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:45.382327    2550 patch_prober.go:28] interesting pod/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 20 20:03:45.382470 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:45.382386    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 20 20:03:46.381803 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:46.381771    2550 patch_prober.go:28] interesting pod/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 20 20:03:46.382300 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:46.382270    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 20 20:03:47.382201 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:47.382159    2550 patch_prober.go:28] interesting pod/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 20 20:03:47.382650 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:47.382215    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 20 20:03:48.381342 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:48.381303    2550 patch_prober.go:28] interesting pod/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 20 20:03:48.381528 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:48.381360    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 20 20:03:49.385130 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:49.385103    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:49.385541 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:49.385450    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:49.386156 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:49.386128    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:03:50.828450 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:50.828401    2550 patch_prober.go:28] interesting pod/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 20 20:03:50.828964 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:50.828465    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 20:03:52.888765 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:52.888733    2550 generic.go:356] "Generic (PLEG): container finished" podID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerID="ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535" exitCode=0
Jul 20 20:03:52.889233 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:52.888814    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" event={"ID":"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab","Type":"ContainerDied","Data":"ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535"}
Jul 20 20:03:52.889319 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:52.889233    2550 scope.go:117] "RemoveContainer" containerID="ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535"
Jul 20 20:03:52.890295 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:52.890242    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:52.890430 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:52.890352    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:52.890430 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:52.890401    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:03:53.895168 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:53.895127    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" event={"ID":"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab","Type":"ContainerStarted","Data":"c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233"}
Jul 20 20:03:54.261228 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:54.261134    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:54.261414 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:54.261357    2550 patch_prober.go:28] interesting pod/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 20 20:03:54.261414 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:54.261405    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 20 20:03:55.261773 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:55.261723    2550 patch_prober.go:28] interesting pod/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 20 20:03:55.262185 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:55.261792    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 20 20:03:56.261225 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:56.261188    2550 patch_prober.go:28] interesting pod/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 20 20:03:56.261448 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:56.261273    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 20 20:03:57.261472 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:57.261426    2550 patch_prober.go:28] interesting pod/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 20 20:03:57.261968 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:57.261489    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 20 20:03:58.261886 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:58.261843    2550 patch_prober.go:28] interesting pod/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 20 20:03:58.262324 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:58.261908    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 20 20:03:59.265768 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:59.265734    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:59.266283 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:59.266247    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:03:59.267123 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:03:59.267101    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:04:03.643871 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.643832    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"]
Jul 20 20:04:03.644484 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.644130    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g" podUID="59d09052-f768-4e65-a420-e7b25c7bbe7e" containerName="maas-api" containerID="cri-o://460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115" gracePeriod=30
Jul 20 20:04:03.901389 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.901316    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:04:03.942575 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.942531    2550 generic.go:356] "Generic (PLEG): container finished" podID="59d09052-f768-4e65-a420-e7b25c7bbe7e" containerID="460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115" exitCode=0
Jul 20 20:04:03.942759 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.942607    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g" event={"ID":"59d09052-f768-4e65-a420-e7b25c7bbe7e","Type":"ContainerDied","Data":"460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115"}
Jul 20 20:04:03.942759 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.942631    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"
Jul 20 20:04:03.942759 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.942658    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g" event={"ID":"59d09052-f768-4e65-a420-e7b25c7bbe7e","Type":"ContainerDied","Data":"b22634178bcc410ee30d84068869796637ee11a9a29b821d452043919d824408"}
Jul 20 20:04:03.942759 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.942688    2550 scope.go:117] "RemoveContainer" containerID="460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115"
Jul 20 20:04:03.952584 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.952556    2550 scope.go:117] "RemoveContainer" containerID="460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115"
Jul 20 20:04:03.952964 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:04:03.952943    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115\": container with ID starting with 460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115 not found: ID does not exist" containerID="460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115"
Jul 20 20:04:03.953015 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.952977    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115"} err="failed to get container status \"460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115\": rpc error: code = NotFound desc = could not find container \"460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115\": container with ID starting with 460fff969fe9c6367bde47795cad9a48a513a6386830555c2cc0e02f50732115 not found: ID does not exist"
Jul 20 20:04:03.990368 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.990330    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d09052-f768-4e65-a420-e7b25c7bbe7e-openshift-service-ca\") pod \"59d09052-f768-4e65-a420-e7b25c7bbe7e\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") "
Jul 20 20:04:03.990550 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.990399    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d09052-f768-4e65-a420-e7b25c7bbe7e-maas-api-tls\") pod \"59d09052-f768-4e65-a420-e7b25c7bbe7e\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") "
Jul 20 20:04:03.990550 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.990432    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x46qq\" (UniqueName: \"kubernetes.io/projected/59d09052-f768-4e65-a420-e7b25c7bbe7e-kube-api-access-x46qq\") pod \"59d09052-f768-4e65-a420-e7b25c7bbe7e\" (UID: \"59d09052-f768-4e65-a420-e7b25c7bbe7e\") "
Jul 20 20:04:03.990729 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.990705    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d09052-f768-4e65-a420-e7b25c7bbe7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59d09052-f768-4e65-a420-e7b25c7bbe7e" (UID: "59d09052-f768-4e65-a420-e7b25c7bbe7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 20:04:03.992648 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.992615    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d09052-f768-4e65-a420-e7b25c7bbe7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59d09052-f768-4e65-a420-e7b25c7bbe7e" (UID: "59d09052-f768-4e65-a420-e7b25c7bbe7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:04:03.992648 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:03.992635    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d09052-f768-4e65-a420-e7b25c7bbe7e-kube-api-access-x46qq" (OuterVolumeSpecName: "kube-api-access-x46qq") pod "59d09052-f768-4e65-a420-e7b25c7bbe7e" (UID: "59d09052-f768-4e65-a420-e7b25c7bbe7e"). InnerVolumeSpecName "kube-api-access-x46qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:04:04.091956 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:04.091917    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59d09052-f768-4e65-a420-e7b25c7bbe7e-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:04.091956 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:04.091953    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59d09052-f768-4e65-a420-e7b25c7bbe7e-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:04.091956 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:04.091964    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46qq\" (UniqueName: \"kubernetes.io/projected/59d09052-f768-4e65-a420-e7b25c7bbe7e-kube-api-access-x46qq\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:04.267319 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:04.267282    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"]
Jul 20 20:04:04.273646 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:04.273613    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-da8568-9f5969759-g4h4g"]
Jul 20 20:04:04.450667 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:04.450630    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59d09052-f768-4e65-a420-e7b25c7bbe7e" path="/var/lib/kubelet/pods/59d09052-f768-4e65-a420-e7b25c7bbe7e/volumes"
Jul 20 20:04:14.796847 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:14.796808    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"]
Jul 20 20:04:14.797277 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:14.797090    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy" containerID="cri-o://a76dbbde1606cdeaa8b1f246588f60f73c3a4eabde6c7a0b7776184c1f2567f4" gracePeriod=30
Jul 20 20:04:19.387918 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:19.387829    2550 patch_prober.go:28] interesting pod/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 20 20:04:19.387918 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:19.387903    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 20:04:20.018626 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.018595    2550 generic.go:356] "Generic (PLEG): container finished" podID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerID="a76dbbde1606cdeaa8b1f246588f60f73c3a4eabde6c7a0b7776184c1f2567f4" exitCode=0
Jul 20 20:04:20.018810 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.018669    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" event={"ID":"a17b32f8-2a78-4a20-8817-4bfbc83fcf38","Type":"ContainerDied","Data":"a76dbbde1606cdeaa8b1f246588f60f73c3a4eabde6c7a0b7776184c1f2567f4"}
Jul 20 20:04:20.018810 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.018766    2550 scope.go:117] "RemoveContainer" containerID="b851d3d3497829181fd2dbdf3203306a003996a92c96120d7e424ae145db6880"
Jul 20 20:04:20.056211 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.056154    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:04:20.130280 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130221    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-certs\") pod \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") "
Jul 20 20:04:20.130280 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130283    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-socket\") pod \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") "
Jul 20 20:04:20.130522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130312    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-envoy\") pod \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") "
Jul 20 20:04:20.130522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130367    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-credential-socket\") pod \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") "
Jul 20 20:04:20.130522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130403    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-podinfo\") pod \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") "
Jul 20 20:04:20.130522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130432    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-data\") pod \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") "
Jul 20 20:04:20.130522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130465    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-token\") pod \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") "
Jul 20 20:04:20.130522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130489    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istiod-ca-cert\") pod \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") "
Jul 20 20:04:20.130522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130521    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm2bn\" (UniqueName: \"kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-kube-api-access-wm2bn\") pod \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\" (UID: \"a17b32f8-2a78-4a20-8817-4bfbc83fcf38\") "
Jul 20 20:04:20.130893 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130620    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a17b32f8-2a78-4a20-8817-4bfbc83fcf38" (UID: "a17b32f8-2a78-4a20-8817-4bfbc83fcf38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:04:20.130893 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130728    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a17b32f8-2a78-4a20-8817-4bfbc83fcf38" (UID: "a17b32f8-2a78-4a20-8817-4bfbc83fcf38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:04:20.130893 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130490    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a17b32f8-2a78-4a20-8817-4bfbc83fcf38" (UID: "a17b32f8-2a78-4a20-8817-4bfbc83fcf38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:04:20.130893 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130848    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:20.130893 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.130868    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-credential-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:20.131197 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.131053    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a17b32f8-2a78-4a20-8817-4bfbc83fcf38" (UID: "a17b32f8-2a78-4a20-8817-4bfbc83fcf38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 20:04:20.131329 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.131309    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-data" (OuterVolumeSpecName: "istio-data") pod "a17b32f8-2a78-4a20-8817-4bfbc83fcf38" (UID: "a17b32f8-2a78-4a20-8817-4bfbc83fcf38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:04:20.133183 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.133156    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a17b32f8-2a78-4a20-8817-4bfbc83fcf38" (UID: "a17b32f8-2a78-4a20-8817-4bfbc83fcf38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:04:20.133335 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.133192    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-kube-api-access-wm2bn" (OuterVolumeSpecName: "kube-api-access-wm2bn") pod "a17b32f8-2a78-4a20-8817-4bfbc83fcf38" (UID: "a17b32f8-2a78-4a20-8817-4bfbc83fcf38"). InnerVolumeSpecName "kube-api-access-wm2bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:04:20.133335 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.133206    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a17b32f8-2a78-4a20-8817-4bfbc83fcf38" (UID: "a17b32f8-2a78-4a20-8817-4bfbc83fcf38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 20:04:20.133335 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.133212    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-token" (OuterVolumeSpecName: "istio-token") pod "a17b32f8-2a78-4a20-8817-4bfbc83fcf38" (UID: "a17b32f8-2a78-4a20-8817-4bfbc83fcf38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:04:20.231318 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.231278    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm2bn\" (UniqueName: \"kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-kube-api-access-wm2bn\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:20.231318 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.231310    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-workload-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:20.231318 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.231321    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-envoy\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:20.231318 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.231328    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-podinfo\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:20.231590 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.231336    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-data\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:20.231590 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.231344    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istio-token\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:20.231590 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:20.231352    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17b32f8-2a78-4a20-8817-4bfbc83fcf38-istiod-ca-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:21.023926 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:21.023897    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"
Jul 20 20:04:21.024427 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:21.023936    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr" event={"ID":"a17b32f8-2a78-4a20-8817-4bfbc83fcf38","Type":"ContainerDied","Data":"e8747ddb9afc75390198622d2bd938b255f51cca319d189c373b5dbc1b48f4fd"}
Jul 20 20:04:21.024427 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:21.023995    2550 scope.go:117] "RemoveContainer" containerID="a76dbbde1606cdeaa8b1f246588f60f73c3a4eabde6c7a0b7776184c1f2567f4"
Jul 20 20:04:21.045898 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:21.045860    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"]
Jul 20 20:04:21.049452 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:21.049428    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-da8568-openshift-default-64885bd7f-wl6wr"]
Jul 20 20:04:22.451270 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:22.451219    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" path="/var/lib/kubelet/pods/a17b32f8-2a78-4a20-8817-4bfbc83fcf38/volumes"
Jul 20 20:04:30.835756 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:30.835720    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"]
Jul 20 20:04:30.836223 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:30.835974    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck" podUID="c8dd5535-70d8-43c9-a13d-2d8ee0ad157f" containerName="maas-api" containerID="cri-o://bc283fbc0f41f38debb0186a56beb4e37503db86ff4f9894b8f5a92b8ada84c8" gracePeriod=30
Jul 20 20:04:31.070243 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.070205    2550 generic.go:356] "Generic (PLEG): container finished" podID="c8dd5535-70d8-43c9-a13d-2d8ee0ad157f" containerID="bc283fbc0f41f38debb0186a56beb4e37503db86ff4f9894b8f5a92b8ada84c8" exitCode=0
Jul 20 20:04:31.070387 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.070251    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck" event={"ID":"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f","Type":"ContainerDied","Data":"bc283fbc0f41f38debb0186a56beb4e37503db86ff4f9894b8f5a92b8ada84c8"}
Jul 20 20:04:31.070387 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.070357    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck" event={"ID":"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f","Type":"ContainerDied","Data":"8c81184a32d41e353cbe37320bccb931c59e42a6eb96708233fb767adefe36b1"}
Jul 20 20:04:31.070387 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.070375    2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c81184a32d41e353cbe37320bccb931c59e42a6eb96708233fb767adefe36b1"
Jul 20 20:04:31.080293 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.080267    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:04:31.104723 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.104647    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-maas-api-tls\") pod \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") "
Jul 20 20:04:31.104869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.104731    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5g8n\" (UniqueName: \"kubernetes.io/projected/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-kube-api-access-w5g8n\") pod \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") "
Jul 20 20:04:31.104869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.104813    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-openshift-service-ca\") pod \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\" (UID: \"c8dd5535-70d8-43c9-a13d-2d8ee0ad157f\") "
Jul 20 20:04:31.105421 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.105378    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8dd5535-70d8-43c9-a13d-2d8ee0ad157f" (UID: "c8dd5535-70d8-43c9-a13d-2d8ee0ad157f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 20:04:31.106912 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.106882    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-kube-api-access-w5g8n" (OuterVolumeSpecName: "kube-api-access-w5g8n") pod "c8dd5535-70d8-43c9-a13d-2d8ee0ad157f" (UID: "c8dd5535-70d8-43c9-a13d-2d8ee0ad157f"). InnerVolumeSpecName "kube-api-access-w5g8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:04:31.107424 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.107402    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8dd5535-70d8-43c9-a13d-2d8ee0ad157f" (UID: "c8dd5535-70d8-43c9-a13d-2d8ee0ad157f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:04:31.205690 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.205654    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5g8n\" (UniqueName: \"kubernetes.io/projected/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-kube-api-access-w5g8n\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:31.205690 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.205682    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:31.205690 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:31.205693    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:32.074932 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:32.074897    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"
Jul 20 20:04:32.095589 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:32.095561    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"]
Jul 20 20:04:32.100045 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:32.100018    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-75e28c-78c94c746d-49nck"]
Jul 20 20:04:32.450113 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:32.450078    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8dd5535-70d8-43c9-a13d-2d8ee0ad157f" path="/var/lib/kubelet/pods/c8dd5535-70d8-43c9-a13d-2d8ee0ad157f/volumes"
Jul 20 20:04:42.079940 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.079901    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"]
Jul 20 20:04:42.080413 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.080106    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy" containerID="cri-o://c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233" gracePeriod=30
Jul 20 20:04:42.972280 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.972220    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"]
Jul 20 20:04:42.972666 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.972649    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy"
Jul 20 20:04:42.972711 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.972669    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy"
Jul 20 20:04:42.972711 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.972685    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8dd5535-70d8-43c9-a13d-2d8ee0ad157f" containerName="maas-api"
Jul 20 20:04:42.972711 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.972691    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8dd5535-70d8-43c9-a13d-2d8ee0ad157f" containerName="maas-api"
Jul 20 20:04:42.972711 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.972705    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d09052-f768-4e65-a420-e7b25c7bbe7e" containerName="maas-api"
Jul 20 20:04:42.972711 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.972711    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d09052-f768-4e65-a420-e7b25c7bbe7e" containerName="maas-api"
Jul 20 20:04:42.972869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.972776    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8dd5535-70d8-43c9-a13d-2d8ee0ad157f" containerName="maas-api"
Jul 20 20:04:42.972869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.972785    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d09052-f768-4e65-a420-e7b25c7bbe7e" containerName="maas-api"
Jul 20 20:04:42.972869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.972792    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy"
Jul 20 20:04:42.974823 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.974807    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:42.977036 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.977016    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-397b55-openshift-default-dockercfg-666p5\""
Jul 20 20:04:42.993430 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:42.993400    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"]
Jul 20 20:04:43.012690 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.012656    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0e78969-7bc0-470a-ad43-8860b4c312eb-istiod-ca-cert\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.012861 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.012817    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-envoy\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.012962 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.012882    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-podinfo\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.012962 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.012911    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-data\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.013100 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.012959    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-certs\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.013100 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.012988    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-token\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.013100 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.013021    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-credential-socket\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.013100 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.013091    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55d74\" (UniqueName: \"kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-kube-api-access-55d74\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.013320 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.013119    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-socket\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.114312 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.114235    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0e78969-7bc0-470a-ad43-8860b4c312eb-istiod-ca-cert\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.114312 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.114318    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-envoy\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.114869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.114360    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-podinfo\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.114869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.114388    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-data\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.114869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.114428    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-certs\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.114869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.114450    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-token\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.114869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.114472    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-credential-socket\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.114869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.114516    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55d74\" (UniqueName: \"kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-kube-api-access-55d74\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.114869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.114544    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-socket\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.115250 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.114970    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-certs\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.115250 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.115000    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-socket\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.115250 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.115044    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0e78969-7bc0-470a-ad43-8860b4c312eb-istiod-ca-cert\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.115250 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.115173    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-data\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.115250 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.115191    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-credential-socket\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.116706 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.116683    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-envoy\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.117009 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.116993    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-podinfo\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.124558 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.124530    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55d74\" (UniqueName: \"kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-kube-api-access-55d74\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.125743 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.125712    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-token\") pod \"e2e-shared-a-397b55-openshift-default-cb5764759-5qj67\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") " pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.285547 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.285451    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:43.427970 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.427940    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"]
Jul 20 20:04:43.429803 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:04:43.429774    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e78969_7bc0_470a_ad43_8860b4c312eb.slice/crio-56b788e86cbf074f798905d1c11f2e89ef35b29f85bc8345aa923835d24b5219 WatchSource:0}: Error finding container 56b788e86cbf074f798905d1c11f2e89ef35b29f85bc8345aa923835d24b5219: Status 404 returned error can't find the container with id 56b788e86cbf074f798905d1c11f2e89ef35b29f85bc8345aa923835d24b5219
Jul 20 20:04:43.431849 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.431816    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:04:43.431931 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.431881    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:04:43.431931 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:43.431910    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:04:44.129725 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:44.129690    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" event={"ID":"f0e78969-7bc0-470a-ad43-8860b4c312eb","Type":"ContainerStarted","Data":"126f34b335ae841097b32ee0ab36c7acdde70d22ac02fe6b32e20f772e51e1de"}
Jul 20 20:04:44.130134 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:44.129783    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" event={"ID":"f0e78969-7bc0-470a-ad43-8860b4c312eb","Type":"ContainerStarted","Data":"56b788e86cbf074f798905d1c11f2e89ef35b29f85bc8345aa923835d24b5219"}
Jul 20 20:04:44.152249 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:44.152195    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" podStartSLOduration=2.152180626 podStartE2EDuration="2.152180626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:04:44.149286298 +0000 UTC m=+2536.235237289" watchObservedRunningTime="2026-07-20 20:04:44.152180626 +0000 UTC m=+2536.238131624"
Jul 20 20:04:44.268724 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:44.268686    2550 patch_prober.go:28] interesting pod/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 20 20:04:44.268905 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:44.268757    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 20 20:04:44.285801 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:44.285765    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:44.290610 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:44.290580    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:45.133582 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:45.133549    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:45.134584 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:45.134565    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:04:46.155373 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.155333    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"]
Jul 20 20:04:46.155869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.155671    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy"
Jul 20 20:04:46.155869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.155685    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy"
Jul 20 20:04:46.155869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.155781    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17b32f8-2a78-4a20-8817-4bfbc83fcf38" containerName="istio-proxy"
Jul 20 20:04:46.157815 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.157792    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:46.162245 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.162218    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-397b55-serving-cert\""
Jul 20 20:04:46.175039 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.175005    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"]
Jul 20 20:04:46.241756 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.241707    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hffkf\" (UniqueName: \"kubernetes.io/projected/cd81587d-00ac-4531-9406-855f481c0272-kube-api-access-hffkf\") pod \"maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:46.241961 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.241873    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd81587d-00ac-4531-9406-855f481c0272-openshift-service-ca\") pod \"maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:46.241961 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.241927    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd81587d-00ac-4531-9406-855f481c0272-maas-api-tls\") pod \"maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:46.342680 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.342634    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd81587d-00ac-4531-9406-855f481c0272-maas-api-tls\") pod \"maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:46.342680 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.342685    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hffkf\" (UniqueName: \"kubernetes.io/projected/cd81587d-00ac-4531-9406-855f481c0272-kube-api-access-hffkf\") pod \"maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:46.342910 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.342746    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd81587d-00ac-4531-9406-855f481c0272-openshift-service-ca\") pod \"maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:46.343460 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.343428    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd81587d-00ac-4531-9406-855f481c0272-openshift-service-ca\") pod \"maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:46.345132 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.345110    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd81587d-00ac-4531-9406-855f481c0272-maas-api-tls\") pod \"maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:46.351059 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.351028    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hffkf\" (UniqueName: \"kubernetes.io/projected/cd81587d-00ac-4531-9406-855f481c0272-kube-api-access-hffkf\") pod \"maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:46.469079 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.468986    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:46.618426 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:46.618395    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"]
Jul 20 20:04:46.620927 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:04:46.620896    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd81587d_00ac_4531_9406_855f481c0272.slice/crio-dbd367d849298c0a69d4c2a8d22283845658e0cfcb92bb71d789641794dd984e WatchSource:0}: Error finding container dbd367d849298c0a69d4c2a8d22283845658e0cfcb92bb71d789641794dd984e: Status 404 returned error can't find the container with id dbd367d849298c0a69d4c2a8d22283845658e0cfcb92bb71d789641794dd984e
Jul 20 20:04:47.144247 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.144209    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l" event={"ID":"cd81587d-00ac-4531-9406-855f481c0272","Type":"ContainerStarted","Data":"9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2"}
Jul 20 20:04:47.144464 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.144279    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l" event={"ID":"cd81587d-00ac-4531-9406-855f481c0272","Type":"ContainerStarted","Data":"dbd367d849298c0a69d4c2a8d22283845658e0cfcb92bb71d789641794dd984e"}
Jul 20 20:04:47.144464 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.144303    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:47.186246 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.186188    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l" podStartSLOduration=0.867523638 podStartE2EDuration="1.186169163s" totalImagesPullingTime="318.645525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:04:47.183788002 +0000 UTC m=+2539.269739012" watchObservedRunningTime="2026-07-20 20:04:47.186169163 +0000 UTC m=+2539.272120161"
Jul 20 20:04:47.327363 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.327333    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:04:47.453291 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453177    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-credential-socket\") pod \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") "
Jul 20 20:04:47.453291 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453252    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-podinfo\") pod \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") "
Jul 20 20:04:47.453616 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453313    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-data\") pod \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") "
Jul 20 20:04:47.453616 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453335    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-envoy\") pod \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") "
Jul 20 20:04:47.453616 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453363    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdzrd\" (UniqueName: \"kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-kube-api-access-bdzrd\") pod \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") "
Jul 20 20:04:47.453616 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453388    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-certs\") pod \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") "
Jul 20 20:04:47.453616 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453420    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istiod-ca-cert\") pod \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") "
Jul 20 20:04:47.453616 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453448    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-token\") pod \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") "
Jul 20 20:04:47.453616 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453469    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-socket\") pod \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\" (UID: \"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab\") "
Jul 20 20:04:47.453616 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453543    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" (UID: "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:04:47.454030 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453650    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" (UID: "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:04:47.454030 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453780    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-credential-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:47.454030 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453801    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:47.454030 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453877    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" (UID: "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:04:47.454030 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.453913    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" (UID: "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 20:04:47.454517 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.454487    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" (UID: "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:04:47.455784 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.455757    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" (UID: "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:04:47.455893 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.455815    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" (UID: "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:04:47.455893 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.455827    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-kube-api-access-bdzrd" (OuterVolumeSpecName: "kube-api-access-bdzrd") pod "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" (UID: "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab"). InnerVolumeSpecName "kube-api-access-bdzrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:04:47.455893 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.455852    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" (UID: "1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 20:04:47.555048 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.555014    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istiod-ca-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:47.555048 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.555043    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-token\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:47.555048 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.555052    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-workload-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:47.555048 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.555061    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-podinfo\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:47.555362 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.555070    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-data\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:47.555362 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.555079    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-istio-envoy\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:47.555362 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:47.555087    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdzrd\" (UniqueName: \"kubernetes.io/projected/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab-kube-api-access-bdzrd\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:04:48.150326 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.150279    2550 generic.go:356] "Generic (PLEG): container finished" podID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerID="c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233" exitCode=0
Jul 20 20:04:48.150515 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.150348    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"
Jul 20 20:04:48.150515 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.150357    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" event={"ID":"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab","Type":"ContainerDied","Data":"c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233"}
Jul 20 20:04:48.150515 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.150437    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29" event={"ID":"1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab","Type":"ContainerDied","Data":"b77075c1e69dd5310da2ab1ac9a7769d7cfdd64dd26f854084459b70bd1a37d4"}
Jul 20 20:04:48.150515 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.150459    2550 scope.go:117] "RemoveContainer" containerID="c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233"
Jul 20 20:04:48.159541 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.159518    2550 scope.go:117] "RemoveContainer" containerID="ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535"
Jul 20 20:04:48.170815 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.170788    2550 scope.go:117] "RemoveContainer" containerID="c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233"
Jul 20 20:04:48.171188 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:04:48.171085    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233\": container with ID starting with c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233 not found: ID does not exist" containerID="c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233"
Jul 20 20:04:48.171188 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.171117    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233"} err="failed to get container status \"c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233\": rpc error: code = NotFound desc = could not find container \"c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233\": container with ID starting with c39e05a6ddbe8ee9a52bbb45d252e0cac143f4633d46c6bd36de618ab63bd233 not found: ID does not exist"
Jul 20 20:04:48.171188 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.171137    2550 scope.go:117] "RemoveContainer" containerID="ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535"
Jul 20 20:04:48.171535 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:04:48.171513    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535\": container with ID starting with ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535 not found: ID does not exist" containerID="ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535"
Jul 20 20:04:48.171585 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.171543    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535"} err="failed to get container status \"ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535\": rpc error: code = NotFound desc = could not find container \"ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535\": container with ID starting with ea5a1ee39fc96d1b460f016be83836918ae85ef8a8dbaa6c64b875bd9a77f535 not found: ID does not exist"
Jul 20 20:04:48.173981 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.173961    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"]
Jul 20 20:04:48.178130 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.178108    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-75e28c-openshift-default-6c648fb989-sbs29"]
Jul 20 20:04:48.451325 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:48.451231    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" path="/var/lib/kubelet/pods/1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab/volumes"
Jul 20 20:04:53.157450 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:53.157412    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:04:54.181500 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.181464    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"]
Jul 20 20:04:54.182861 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.182835    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy"
Jul 20 20:04:54.183045 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.183032    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy"
Jul 20 20:04:54.183163 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.183153    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy"
Jul 20 20:04:54.183242 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.183233    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy"
Jul 20 20:04:54.183522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.183508    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy"
Jul 20 20:04:54.183615 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.183604    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd2af0c-6b2c-4f0d-a008-dba2da7ff6ab" containerName="istio-proxy"
Jul 20 20:04:54.187286 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.187232    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:54.191003 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.190980    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-6fb9c1-serving-cert\""
Jul 20 20:04:54.199739 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.199710    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"]
Jul 20 20:04:54.314964 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.314926    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbp2p\" (UniqueName: \"kubernetes.io/projected/9b7ac406-3972-4bca-a827-2ebab0cc57a3-kube-api-access-fbp2p\") pod \"maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:54.314964 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.314966    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7ac406-3972-4bca-a827-2ebab0cc57a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:54.315190 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.315007    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7ac406-3972-4bca-a827-2ebab0cc57a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:54.415848 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.415800    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7ac406-3972-4bca-a827-2ebab0cc57a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:54.416056 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.415912    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp2p\" (UniqueName: \"kubernetes.io/projected/9b7ac406-3972-4bca-a827-2ebab0cc57a3-kube-api-access-fbp2p\") pod \"maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:54.416056 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.415939    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7ac406-3972-4bca-a827-2ebab0cc57a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:54.416531 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.416502    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7ac406-3972-4bca-a827-2ebab0cc57a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:54.418522 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.418501    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7ac406-3972-4bca-a827-2ebab0cc57a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:54.427499 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.427465    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbp2p\" (UniqueName: \"kubernetes.io/projected/9b7ac406-3972-4bca-a827-2ebab0cc57a3-kube-api-access-fbp2p\") pod \"maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:54.502156 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.502053    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:54.698074 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:54.698046    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"]
Jul 20 20:04:54.699741 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:04:54.699705    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7ac406_3972_4bca_a827_2ebab0cc57a3.slice/crio-e4c942cfd215bd326ff3a6b3133f3ca6483a1d68383e6981b4dd26e8ba0bef51 WatchSource:0}: Error finding container e4c942cfd215bd326ff3a6b3133f3ca6483a1d68383e6981b4dd26e8ba0bef51: Status 404 returned error can't find the container with id e4c942cfd215bd326ff3a6b3133f3ca6483a1d68383e6981b4dd26e8ba0bef51
Jul 20 20:04:55.187530 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:55.187491    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf" event={"ID":"9b7ac406-3972-4bca-a827-2ebab0cc57a3","Type":"ContainerStarted","Data":"66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e"}
Jul 20 20:04:55.187930 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:55.187551    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf" event={"ID":"9b7ac406-3972-4bca-a827-2ebab0cc57a3","Type":"ContainerStarted","Data":"e4c942cfd215bd326ff3a6b3133f3ca6483a1d68383e6981b4dd26e8ba0bef51"}
Jul 20 20:04:55.187930 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:55.187579    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:04:55.208778 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:04:55.208724    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf" podStartSLOduration=0.859735343 podStartE2EDuration="1.208708632s" totalImagesPullingTime="348.973289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:04:55.208393275 +0000 UTC m=+2547.294344293" watchObservedRunningTime="2026-07-20 20:04:55.208708632 +0000 UTC m=+2547.294659629"
Jul 20 20:05:01.174453 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:01.174412    2550 patch_prober.go:28] interesting pod/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 20 20:05:01.175695 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:01.174493    2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 20 20:05:01.197142 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:01.197112    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:05:02.220874 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:02.220842    2550 generic.go:356] "Generic (PLEG): container finished" podID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerID="126f34b335ae841097b32ee0ab36c7acdde70d22ac02fe6b32e20f772e51e1de" exitCode=0
Jul 20 20:05:02.220874 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:02.220884    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" event={"ID":"f0e78969-7bc0-470a-ad43-8860b4c312eb","Type":"ContainerDied","Data":"126f34b335ae841097b32ee0ab36c7acdde70d22ac02fe6b32e20f772e51e1de"}
Jul 20 20:05:02.221335 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:02.221238    2550 scope.go:117] "RemoveContainer" containerID="126f34b335ae841097b32ee0ab36c7acdde70d22ac02fe6b32e20f772e51e1de"
Jul 20 20:05:02.222225 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:02.222195    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:05:02.222295 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:02.222282    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:05:02.222336 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:02.222315    2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 20 20:05:03.232463 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:03.232411    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" event={"ID":"f0e78969-7bc0-470a-ad43-8860b4c312eb","Type":"ContainerStarted","Data":"b5bfbbed26b8627a81995cce4461925683adda61e9bd57bfabd8ecec19eeb46a"}
Jul 20 20:05:03.285835 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:03.285794    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:05:03.286073 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:03.286047    2550 patch_prober.go:28] interesting pod/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 20 20:05:03.286158 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:03.286105    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 20 20:05:04.285928 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:04.285886    2550 patch_prober.go:28] interesting pod/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 20 20:05:04.286446 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:04.285952    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 20 20:05:05.285925 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:05.285887    2550 patch_prober.go:28] interesting pod/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 20 20:05:05.286109 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:05.285963    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 20 20:05:06.286671 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:06.286631    2550 patch_prober.go:28] interesting pod/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 20 20:05:06.287081 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:06.286689    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 20 20:05:07.285894 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:07.285854    2550 patch_prober.go:28] interesting pod/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 20 20:05:07.286099 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:07.285911    2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 20 20:05:08.289973 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:08.289942    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:05:08.290402 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:08.290349    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:05:08.291222 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:08.291200    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:05:39.316561 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.316529    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"]
Jul 20 20:05:39.318943 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.318926    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.321794 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.321770    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-397b55\"/\"auth-test-model-397b55-kserve-self-signed-certs\""
Jul 20 20:05:39.321794 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.321768    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-397b55\"/\"default-dockercfg-bfpcq\""
Jul 20 20:05:39.322028 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.321770    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-397b55\"/\"kube-root-ca.crt\""
Jul 20 20:05:39.322028 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.321818    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-397b55\"/\"openshift-service-ca.crt\""
Jul 20 20:05:39.326816 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.326791    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"]
Jul 20 20:05:39.442786 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.442735    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tls-certs\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.442786 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.442793    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-dshm\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.443031 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.442848    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tmp-dir\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.443031 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.442885    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-model-cache\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.443031 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.442915    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-home\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.443031 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.442942    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66rn\" (UniqueName: \"kubernetes.io/projected/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-kube-api-access-d66rn\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.543651 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.543611    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-model-cache\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.543821 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.543668    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-home\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.543821 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.543706    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66rn\" (UniqueName: \"kubernetes.io/projected/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-kube-api-access-d66rn\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.543821 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.543769    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tls-certs\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.544099 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.544075    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-dshm\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.544099 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.544089    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-model-cache\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.544236 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.544120    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-home\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.544236 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.544130    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tmp-dir\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.544430 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.544413    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tmp-dir\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.546106 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.546085    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-dshm\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.546378 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.546360    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tls-certs\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.555471 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.555450    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66rn\" (UniqueName: \"kubernetes.io/projected/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-kube-api-access-d66rn\") pod \"auth-test-model-397b55-kserve-5f444dfb7f-k5hw6\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") " pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.629104 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.629072    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:39.763051 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:39.763025    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"]
Jul 20 20:05:39.767640 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:05:39.766362    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa2891e5_8e9a_4fc4_9820_d943f078f8d0.slice/crio-d5183ca8efe4be862f4a00756de2da4b0db15092b99d3ddee95b60d664a4c4c0 WatchSource:0}: Error finding container d5183ca8efe4be862f4a00756de2da4b0db15092b99d3ddee95b60d664a4c4c0: Status 404 returned error can't find the container with id d5183ca8efe4be862f4a00756de2da4b0db15092b99d3ddee95b60d664a4c4c0
Jul 20 20:05:40.402408 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:40.402369    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6" event={"ID":"fa2891e5-8e9a-4fc4-9820-d943f078f8d0","Type":"ContainerStarted","Data":"a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73"}
Jul 20 20:05:40.402815 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:40.402413    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6" event={"ID":"fa2891e5-8e9a-4fc4-9820-d943f078f8d0","Type":"ContainerStarted","Data":"d5183ca8efe4be862f4a00756de2da4b0db15092b99d3ddee95b60d664a4c4c0"}
Jul 20 20:05:40.427341 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:40.427287    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6" podStartSLOduration=1.427271151 podStartE2EDuration="1.427271151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:05:40.424013417 +0000 UTC m=+2592.509964416" watchObservedRunningTime="2026-07-20 20:05:40.427271151 +0000 UTC m=+2592.513222139"
Jul 20 20:05:49.630228 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:49.630129    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:49.630228 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:49.630198    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:49.642957 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:49.642932    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:05:50.458869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:05:50.458836    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:06:04.024622 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.024582    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"]
Jul 20 20:06:04.027435 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.027419    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.030484 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.030462    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"openshift-service-ca.crt\""
Jul 20 20:06:04.031487 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.031464    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"auth-test-model-6fb9c1-kserve-self-signed-certs\""
Jul 20 20:06:04.031601 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.031484    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"kube-root-ca.crt\""
Jul 20 20:06:04.031601 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.031528    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"default-dockercfg-w2vrt\""
Jul 20 20:06:04.038448 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.038427    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"]
Jul 20 20:06:04.058362 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.058333    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn4mm\" (UniqueName: \"kubernetes.io/projected/3a2fb36b-6604-4e89-9a0d-7834da1b0373-kube-api-access-pn4mm\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.058545 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.058370    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tls-certs\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.058545 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.058399    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-home\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.058545 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.058462    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tmp-dir\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.058545 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.058508    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-dshm\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.058545 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.058527    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-model-cache\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.159693 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.159653    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-dshm\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.159693 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.159693    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-model-cache\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.159937 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.159742    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn4mm\" (UniqueName: \"kubernetes.io/projected/3a2fb36b-6604-4e89-9a0d-7834da1b0373-kube-api-access-pn4mm\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.159937 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.159766    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tls-certs\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.159937 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.159806    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-home\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.160099 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.159951    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tmp-dir\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.160212 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.160189    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-model-cache\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.160294 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.160211    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-home\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.160357 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.160292    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tmp-dir\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.161983 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.161961    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-dshm\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.162322 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.162303    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tls-certs\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.168021 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.168000    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn4mm\" (UniqueName: \"kubernetes.io/projected/3a2fb36b-6604-4e89-9a0d-7834da1b0373-kube-api-access-pn4mm\") pod \"auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") " pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.337984 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.337886    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:04.483369 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.483204    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"]
Jul 20 20:06:04.485791 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:06:04.485765    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a2fb36b_6604_4e89_9a0d_7834da1b0373.slice/crio-44bc1aa46b1b8c2bf369d4a6595a822bead7b64656fad3f39cdb6ca15a3acf54 WatchSource:0}: Error finding container 44bc1aa46b1b8c2bf369d4a6595a822bead7b64656fad3f39cdb6ca15a3acf54: Status 404 returned error can't find the container with id 44bc1aa46b1b8c2bf369d4a6595a822bead7b64656fad3f39cdb6ca15a3acf54
Jul 20 20:06:04.511561 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:04.511525    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx" event={"ID":"3a2fb36b-6604-4e89-9a0d-7834da1b0373","Type":"ContainerStarted","Data":"44bc1aa46b1b8c2bf369d4a6595a822bead7b64656fad3f39cdb6ca15a3acf54"}
Jul 20 20:06:05.517975 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:05.517935    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx" event={"ID":"3a2fb36b-6604-4e89-9a0d-7834da1b0373","Type":"ContainerStarted","Data":"2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061"}
Jul 20 20:06:05.537709 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:05.537651    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx" podStartSLOduration=1.5376360999999998 podStartE2EDuration="1.5376361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:06:05.535828682 +0000 UTC m=+2617.621779681" watchObservedRunningTime="2026-07-20 20:06:05.5376361 +0000 UTC m=+2617.623587146"
Jul 20 20:06:14.338583 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:14.338538    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:14.339196 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:14.338610    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:14.351233 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:14.351201    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:06:14.569758 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:06:14.569726    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:07:28.521782 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:07:28.521684    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 20:07:28.533404 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:07:28.533366    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 20:07:28.702378 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:07:28.702342    2550 scope.go:117] "RemoveContainer" containerID="889073801f208d1882e65b04315131909be58d078decf79684cfa5bf094d9c2a"
Jul 20 20:08:06.451591 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:06.451557    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"]
Jul 20 20:08:07.116452 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:07.116418    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"]
Jul 20 20:08:10.621637 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.621601    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"]
Jul 20 20:08:10.625668 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.624940    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.631488 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.631460    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-397b55\"/\"sub-test-model-397b55-kserve-self-signed-certs\""
Jul 20 20:08:10.645067 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.645029    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"]
Jul 20 20:08:10.791872 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.791810    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0558e524-d852-49bc-ae78-a02b954f27b3-tls-certs\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.792066 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.791926    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-home\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.792066 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.791965    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-dshm\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.792066 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.791985    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtphs\" (UniqueName: \"kubernetes.io/projected/0558e524-d852-49bc-ae78-a02b954f27b3-kube-api-access-wtphs\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.792180 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.792080    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-model-cache\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.792180 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.792112    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-tmp-dir\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.893335 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.893221    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-model-cache\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.893335 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.893302    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-tmp-dir\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.893533 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.893355    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0558e524-d852-49bc-ae78-a02b954f27b3-tls-certs\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.893533 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.893425    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-home\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.893606 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.893556    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-dshm\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.893656 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.893604    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtphs\" (UniqueName: \"kubernetes.io/projected/0558e524-d852-49bc-ae78-a02b954f27b3-kube-api-access-wtphs\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.893763 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.893739    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-model-cache\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.893838 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.893758    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-tmp-dir\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.893838 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.893797    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-home\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.895833 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.895803    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-dshm\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.896087 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.896070    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0558e524-d852-49bc-ae78-a02b954f27b3-tls-certs\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.903661 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.903631    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtphs\" (UniqueName: \"kubernetes.io/projected/0558e524-d852-49bc-ae78-a02b954f27b3-kube-api-access-wtphs\") pod \"sub-test-model-397b55-kserve-6f57f4c75c-qdfrn\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") " pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:10.936490 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:10.936449    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:11.075673 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:11.075638    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"]
Jul 20 20:08:11.081045 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:08:11.081009    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0558e524_d852_49bc_ae78_a02b954f27b3.slice/crio-a23740283327f1b2cf840e587894a2c5930db95df2bb4055d302aa127a009190 WatchSource:0}: Error finding container a23740283327f1b2cf840e587894a2c5930db95df2bb4055d302aa127a009190: Status 404 returned error can't find the container with id a23740283327f1b2cf840e587894a2c5930db95df2bb4055d302aa127a009190
Jul 20 20:08:11.098302 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:11.098276    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn" event={"ID":"0558e524-d852-49bc-ae78-a02b954f27b3","Type":"ContainerStarted","Data":"a23740283327f1b2cf840e587894a2c5930db95df2bb4055d302aa127a009190"}
Jul 20 20:08:12.104216 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:12.104171    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn" event={"ID":"0558e524-d852-49bc-ae78-a02b954f27b3","Type":"ContainerStarted","Data":"a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5"}
Jul 20 20:08:12.124650 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:12.124603    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn" podStartSLOduration=2.124587515 podStartE2EDuration="2.124587515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:08:12.122934172 +0000 UTC m=+2744.208885171" watchObservedRunningTime="2026-07-20 20:08:12.124587515 +0000 UTC m=+2744.210538512"
Jul 20 20:08:20.936762 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:20.936724    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:20.936762 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:20.936770    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:20.949548 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:20.949520    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:21.155161 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.155131    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:08:21.476014 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.475974    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6" podUID="fa2891e5-8e9a-4fc4-9820-d943f078f8d0" containerName="main" containerID="cri-o://a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73" gracePeriod=45
Jul 20 20:08:21.728608 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.728549    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:08:21.786293 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.786243    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d66rn\" (UniqueName: \"kubernetes.io/projected/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-kube-api-access-d66rn\") pod \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") "
Jul 20 20:08:21.786435 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.786305    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tls-certs\") pod \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") "
Jul 20 20:08:21.786435 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.786357    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-model-cache\") pod \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") "
Jul 20 20:08:21.786435 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.786407    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-dshm\") pod \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") "
Jul 20 20:08:21.786549 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.786438    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tmp-dir\") pod \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") "
Jul 20 20:08:21.786549 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.786517    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-home\") pod \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\" (UID: \"fa2891e5-8e9a-4fc4-9820-d943f078f8d0\") "
Jul 20 20:08:21.786703 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.786655    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa2891e5-8e9a-4fc4-9820-d943f078f8d0" (UID: "fa2891e5-8e9a-4fc4-9820-d943f078f8d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:08:21.786827 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.786695    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa2891e5-8e9a-4fc4-9820-d943f078f8d0" (UID: "fa2891e5-8e9a-4fc4-9820-d943f078f8d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:08:21.786827 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.786789    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-model-cache\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:21.786827 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.786786    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-home" (OuterVolumeSpecName: "home") pod "fa2891e5-8e9a-4fc4-9820-d943f078f8d0" (UID: "fa2891e5-8e9a-4fc4-9820-d943f078f8d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:08:21.786827 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.786801    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tmp-dir\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:21.788554 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.788531    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-kube-api-access-d66rn" (OuterVolumeSpecName: "kube-api-access-d66rn") pod "fa2891e5-8e9a-4fc4-9820-d943f078f8d0" (UID: "fa2891e5-8e9a-4fc4-9820-d943f078f8d0"). InnerVolumeSpecName "kube-api-access-d66rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:08:21.788894 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.788876    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-dshm" (OuterVolumeSpecName: "dshm") pod "fa2891e5-8e9a-4fc4-9820-d943f078f8d0" (UID: "fa2891e5-8e9a-4fc4-9820-d943f078f8d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:08:21.788953 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.788892    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa2891e5-8e9a-4fc4-9820-d943f078f8d0" (UID: "fa2891e5-8e9a-4fc4-9820-d943f078f8d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:08:21.887781 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.887747    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-home\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:21.887781 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.887776    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d66rn\" (UniqueName: \"kubernetes.io/projected/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-kube-api-access-d66rn\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:21.887781 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.887785    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-tls-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:21.887781 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:21.887793    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa2891e5-8e9a-4fc4-9820-d943f078f8d0-dshm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:22.136049 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.135995    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx" podUID="3a2fb36b-6604-4e89-9a0d-7834da1b0373" containerName="main" containerID="cri-o://2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061" gracePeriod=45
Jul 20 20:08:22.149669 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.149643    2550 generic.go:356] "Generic (PLEG): container finished" podID="fa2891e5-8e9a-4fc4-9820-d943f078f8d0" containerID="a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73" exitCode=0
Jul 20 20:08:22.149795 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.149685    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"
Jul 20 20:08:22.149795 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.149725    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6" event={"ID":"fa2891e5-8e9a-4fc4-9820-d943f078f8d0","Type":"ContainerDied","Data":"a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73"}
Jul 20 20:08:22.149795 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.149770    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6" event={"ID":"fa2891e5-8e9a-4fc4-9820-d943f078f8d0","Type":"ContainerDied","Data":"d5183ca8efe4be862f4a00756de2da4b0db15092b99d3ddee95b60d664a4c4c0"}
Jul 20 20:08:22.149795 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.149788    2550 scope.go:117] "RemoveContainer" containerID="a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73"
Jul 20 20:08:22.201799 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.201763    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"]
Jul 20 20:08:22.208775 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.208749    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/auth-test-model-397b55-kserve-5f444dfb7f-k5hw6"]
Jul 20 20:08:22.246792 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.246769    2550 scope.go:117] "RemoveContainer" containerID="a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73"
Jul 20 20:08:22.247106 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:08:22.247087    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73\": container with ID starting with a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73 not found: ID does not exist" containerID="a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73"
Jul 20 20:08:22.247191 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.247112    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73"} err="failed to get container status \"a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73\": rpc error: code = NotFound desc = could not find container \"a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73\": container with ID starting with a6120c5a01e40dba2c4cc2124a7bf895147910e9a452b055ea12ca6187af9b73 not found: ID does not exist"
Jul 20 20:08:22.383890 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.383864    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:08:22.391557 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.391503    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-dshm\") pod \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") "
Jul 20 20:08:22.391557 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.391550    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn4mm\" (UniqueName: \"kubernetes.io/projected/3a2fb36b-6604-4e89-9a0d-7834da1b0373-kube-api-access-pn4mm\") pod \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") "
Jul 20 20:08:22.391744 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.391568    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-home\") pod \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") "
Jul 20 20:08:22.391744 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.391592    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tmp-dir\") pod \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") "
Jul 20 20:08:22.391744 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.391652    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tls-certs\") pod \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") "
Jul 20 20:08:22.391744 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.391740    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-model-cache\") pod \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\" (UID: \"3a2fb36b-6604-4e89-9a0d-7834da1b0373\") "
Jul 20 20:08:22.391942 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.391825    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-home" (OuterVolumeSpecName: "home") pod "3a2fb36b-6604-4e89-9a0d-7834da1b0373" (UID: "3a2fb36b-6604-4e89-9a0d-7834da1b0373"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:08:22.391942 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.391836    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a2fb36b-6604-4e89-9a0d-7834da1b0373" (UID: "3a2fb36b-6604-4e89-9a0d-7834da1b0373"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:08:22.392058 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.391944    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a2fb36b-6604-4e89-9a0d-7834da1b0373" (UID: "3a2fb36b-6604-4e89-9a0d-7834da1b0373"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:08:22.392058 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.392004    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-home\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:22.392058 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.392018    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tmp-dir\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:22.393770 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.393745    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-dshm" (OuterVolumeSpecName: "dshm") pod "3a2fb36b-6604-4e89-9a0d-7834da1b0373" (UID: "3a2fb36b-6604-4e89-9a0d-7834da1b0373"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:08:22.393873 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.393789    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2fb36b-6604-4e89-9a0d-7834da1b0373-kube-api-access-pn4mm" (OuterVolumeSpecName: "kube-api-access-pn4mm") pod "3a2fb36b-6604-4e89-9a0d-7834da1b0373" (UID: "3a2fb36b-6604-4e89-9a0d-7834da1b0373"). InnerVolumeSpecName "kube-api-access-pn4mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:08:22.393873 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.393808    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a2fb36b-6604-4e89-9a0d-7834da1b0373" (UID: "3a2fb36b-6604-4e89-9a0d-7834da1b0373"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:08:22.450713 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.450680    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa2891e5-8e9a-4fc4-9820-d943f078f8d0" path="/var/lib/kubelet/pods/fa2891e5-8e9a-4fc4-9820-d943f078f8d0/volumes"
Jul 20 20:08:22.493481 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.493446    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-dshm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:22.493481 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.493481    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn4mm\" (UniqueName: \"kubernetes.io/projected/3a2fb36b-6604-4e89-9a0d-7834da1b0373-kube-api-access-pn4mm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:22.493729 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.493496    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a2fb36b-6604-4e89-9a0d-7834da1b0373-tls-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:22.493729 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:22.493509    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a2fb36b-6604-4e89-9a0d-7834da1b0373-model-cache\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:08:23.155944 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:23.155912    2550 generic.go:356] "Generic (PLEG): container finished" podID="3a2fb36b-6604-4e89-9a0d-7834da1b0373" containerID="2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061" exitCode=0
Jul 20 20:08:23.156421 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:23.155951    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"
Jul 20 20:08:23.156421 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:23.155993    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx" event={"ID":"3a2fb36b-6604-4e89-9a0d-7834da1b0373","Type":"ContainerDied","Data":"2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061"}
Jul 20 20:08:23.156421 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:23.156032    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx" event={"ID":"3a2fb36b-6604-4e89-9a0d-7834da1b0373","Type":"ContainerDied","Data":"44bc1aa46b1b8c2bf369d4a6595a822bead7b64656fad3f39cdb6ca15a3acf54"}
Jul 20 20:08:23.156421 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:23.156050    2550 scope.go:117] "RemoveContainer" containerID="2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061"
Jul 20 20:08:23.164166 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:23.164147    2550 scope.go:117] "RemoveContainer" containerID="2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061"
Jul 20 20:08:23.164444 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:08:23.164426    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061\": container with ID starting with 2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061 not found: ID does not exist" containerID="2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061"
Jul 20 20:08:23.164530 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:23.164452    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061"} err="failed to get container status \"2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061\": rpc error: code = NotFound desc = could not find container \"2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061\": container with ID starting with 2abab2d12e46637a5c98bd91972c8147b1af27590d437d573efcf4b4b0b9e061 not found: ID does not exist"
Jul 20 20:08:23.179194 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:23.179166    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"]
Jul 20 20:08:23.185749 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:23.185722    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/auth-test-model-6fb9c1-kserve-6f6496c459-xs2rx"]
Jul 20 20:08:24.449787 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:24.449750    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a2fb36b-6604-4e89-9a0d-7834da1b0373" path="/var/lib/kubelet/pods/3a2fb36b-6604-4e89-9a0d-7834da1b0373/volumes"
Jul 20 20:08:41.450531 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.450492    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"]
Jul 20 20:08:41.451013 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.450823    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa2891e5-8e9a-4fc4-9820-d943f078f8d0" containerName="main"
Jul 20 20:08:41.451013 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.450836    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa2891e5-8e9a-4fc4-9820-d943f078f8d0" containerName="main"
Jul 20 20:08:41.451013 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.450851    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2fb36b-6604-4e89-9a0d-7834da1b0373" containerName="main"
Jul 20 20:08:41.451013 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.450856    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2fb36b-6604-4e89-9a0d-7834da1b0373" containerName="main"
Jul 20 20:08:41.451013 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.450911    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a2fb36b-6604-4e89-9a0d-7834da1b0373" containerName="main"
Jul 20 20:08:41.451013 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.450920    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa2891e5-8e9a-4fc4-9820-d943f078f8d0" containerName="main"
Jul 20 20:08:41.453828 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.453809    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.456242 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.456195    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"kube-root-ca.crt\""
Jul 20 20:08:41.457216 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.457192    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"openshift-service-ca.crt\""
Jul 20 20:08:41.457395 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.457304    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"default-dockercfg-w2vrt\""
Jul 20 20:08:41.457395 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.457315    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"sub-test-model-6fb9c1-kserve-self-signed-certs\""
Jul 20 20:08:41.467856 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.467829    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"]
Jul 20 20:08:41.557869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.557830    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-model-cache\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.558067 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.557965    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vgjt\" (UniqueName: \"kubernetes.io/projected/12070920-2ff6-472a-9bc6-006232430ab5-kube-api-access-6vgjt\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.558067 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.558032    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12070920-2ff6-472a-9bc6-006232430ab5-tls-certs\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.558186 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.558074    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-tmp-dir\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.558186 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.558112    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-home\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.558186 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.558158    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-dshm\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.659408 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.659371    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-home\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.659408 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.659411    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-dshm\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.659655 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.659440    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-model-cache\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.659655 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.659517    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgjt\" (UniqueName: \"kubernetes.io/projected/12070920-2ff6-472a-9bc6-006232430ab5-kube-api-access-6vgjt\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.659655 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.659550    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12070920-2ff6-472a-9bc6-006232430ab5-tls-certs\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.659655 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.659582    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-tmp-dir\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.659922 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.659898    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-model-cache\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.659973 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.659921    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-home\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.659973 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.659946    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-tmp-dir\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.661845 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.661815    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-dshm\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.662042 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.662027    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12070920-2ff6-472a-9bc6-006232430ab5-tls-certs\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.671091 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.671071    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgjt\" (UniqueName: \"kubernetes.io/projected/12070920-2ff6-472a-9bc6-006232430ab5-kube-api-access-6vgjt\") pod \"sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") " pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.763145 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.763066    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:41.897190 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:41.897165    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"]
Jul 20 20:08:41.899199 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:08:41.899168    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12070920_2ff6_472a_9bc6_006232430ab5.slice/crio-c2e3a7bdb17d2ed6b592ee7498cc35f1b8bd1768385800e4c55d2b928361f145 WatchSource:0}: Error finding container c2e3a7bdb17d2ed6b592ee7498cc35f1b8bd1768385800e4c55d2b928361f145: Status 404 returned error can't find the container with id c2e3a7bdb17d2ed6b592ee7498cc35f1b8bd1768385800e4c55d2b928361f145
Jul 20 20:08:42.242879 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:42.242843    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr" event={"ID":"12070920-2ff6-472a-9bc6-006232430ab5","Type":"ContainerStarted","Data":"e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195"}
Jul 20 20:08:42.242879 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:42.242885    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr" event={"ID":"12070920-2ff6-472a-9bc6-006232430ab5","Type":"ContainerStarted","Data":"c2e3a7bdb17d2ed6b592ee7498cc35f1b8bd1768385800e4c55d2b928361f145"}
Jul 20 20:08:42.265198 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:42.265147    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr" podStartSLOduration=1.265131684 podStartE2EDuration="1.265131684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:08:42.262066304 +0000 UTC m=+2774.348017305" watchObservedRunningTime="2026-07-20 20:08:42.265131684 +0000 UTC m=+2774.351082682"
Jul 20 20:08:51.763369 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:51.763242    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:51.763369 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:51.763320    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:51.776388 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:51.776360    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:08:52.297511 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:08:52.297483    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:09:46.853924 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:46.853891    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"]
Jul 20 20:09:47.625523 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:47.625489    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"]
Jul 20 20:09:50.232236 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.232195    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"]
Jul 20 20:09:50.235810 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.235786    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.238452 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.238433    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-397b55\"/\"rate-test-model-397b55-kserve-self-signed-certs\""
Jul 20 20:09:50.245560 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.245531    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"]
Jul 20 20:09:50.367107 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.367062    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16aba4c9-0e95-44fb-8029-833e4051d251-tls-certs\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.367107 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.367112    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-model-cache\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.367366 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.367182    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdl2x\" (UniqueName: \"kubernetes.io/projected/16aba4c9-0e95-44fb-8029-833e4051d251-kube-api-access-qdl2x\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.367366 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.367228    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-home\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.367366 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.367287    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-dshm\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.367366 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.367311    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-tmp-dir\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.468122 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.468091    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16aba4c9-0e95-44fb-8029-833e4051d251-tls-certs\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.468122 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.468123    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-model-cache\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.468403 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.468149    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdl2x\" (UniqueName: \"kubernetes.io/projected/16aba4c9-0e95-44fb-8029-833e4051d251-kube-api-access-qdl2x\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.468403 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.468176    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-home\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.468403 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.468213    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-dshm\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.468403 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.468241    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-tmp-dir\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.468625 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.468548    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-model-cache\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.468625 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.468593    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-home\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.468709 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.468601    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-tmp-dir\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.470425 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.470407    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-dshm\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.470767 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.470750    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16aba4c9-0e95-44fb-8029-833e4051d251-tls-certs\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.483052 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.482986    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdl2x\" (UniqueName: \"kubernetes.io/projected/16aba4c9-0e95-44fb-8029-833e4051d251-kube-api-access-qdl2x\") pod \"rate-test-model-397b55-kserve-5f87487d95-rwjvl\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") " pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.545558 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.545527    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:09:50.679352 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:50.679326    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"]
Jul 20 20:09:50.681193 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:09:50.681163    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16aba4c9_0e95_44fb_8029_833e4051d251.slice/crio-7d6ec8a9ee241d4f9ed9c99132a3cdf6125c4c239f1c70b886b90ed70446f00e WatchSource:0}: Error finding container 7d6ec8a9ee241d4f9ed9c99132a3cdf6125c4c239f1c70b886b90ed70446f00e: Status 404 returned error can't find the container with id 7d6ec8a9ee241d4f9ed9c99132a3cdf6125c4c239f1c70b886b90ed70446f00e
Jul 20 20:09:51.553571 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:51.553534    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl" event={"ID":"16aba4c9-0e95-44fb-8029-833e4051d251","Type":"ContainerStarted","Data":"4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd"}
Jul 20 20:09:51.553571 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:51.553579    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl" event={"ID":"16aba4c9-0e95-44fb-8029-833e4051d251","Type":"ContainerStarted","Data":"7d6ec8a9ee241d4f9ed9c99132a3cdf6125c4c239f1c70b886b90ed70446f00e"}
Jul 20 20:09:51.571156 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:09:51.571109    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl" podStartSLOduration=1.571093466 podStartE2EDuration="1.571093466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:09:51.570627478 +0000 UTC m=+2843.656578474" watchObservedRunningTime="2026-07-20 20:09:51.571093466 +0000 UTC m=+2843.657044465"
Jul 20 20:10:00.546219 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:00.546178    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:10:00.546219 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:00.546230    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:10:00.558690 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:00.558667    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:10:00.608897 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:00.608870    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:10:01.873199 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:01.873159    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn" podUID="0558e524-d852-49bc-ae78-a02b954f27b3" containerName="main" containerID="cri-o://a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5" gracePeriod=45
Jul 20 20:10:02.120419 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.120389    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:10:02.176490 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.176407    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-model-cache\") pod \"0558e524-d852-49bc-ae78-a02b954f27b3\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") "
Jul 20 20:10:02.176654 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.176513    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0558e524-d852-49bc-ae78-a02b954f27b3-tls-certs\") pod \"0558e524-d852-49bc-ae78-a02b954f27b3\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") "
Jul 20 20:10:02.176654 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.176546    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-home\") pod \"0558e524-d852-49bc-ae78-a02b954f27b3\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") "
Jul 20 20:10:02.176654 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.176573    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-tmp-dir\") pod \"0558e524-d852-49bc-ae78-a02b954f27b3\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") "
Jul 20 20:10:02.176654 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.176625    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtphs\" (UniqueName: \"kubernetes.io/projected/0558e524-d852-49bc-ae78-a02b954f27b3-kube-api-access-wtphs\") pod \"0558e524-d852-49bc-ae78-a02b954f27b3\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") "
Jul 20 20:10:02.176881 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.176663    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "0558e524-d852-49bc-ae78-a02b954f27b3" (UID: "0558e524-d852-49bc-ae78-a02b954f27b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:10:02.176881 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.176676    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-dshm\") pod \"0558e524-d852-49bc-ae78-a02b954f27b3\" (UID: \"0558e524-d852-49bc-ae78-a02b954f27b3\") "
Jul 20 20:10:02.176881 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.176778    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-home" (OuterVolumeSpecName: "home") pod "0558e524-d852-49bc-ae78-a02b954f27b3" (UID: "0558e524-d852-49bc-ae78-a02b954f27b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:10:02.177014 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.176919    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0558e524-d852-49bc-ae78-a02b954f27b3" (UID: "0558e524-d852-49bc-ae78-a02b954f27b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:10:02.177014 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.176989    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-home\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:02.177014 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.177008    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-model-cache\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:02.178736 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.178711    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0558e524-d852-49bc-ae78-a02b954f27b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0558e524-d852-49bc-ae78-a02b954f27b3" (UID: "0558e524-d852-49bc-ae78-a02b954f27b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:10:02.179187 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.179157    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0558e524-d852-49bc-ae78-a02b954f27b3-kube-api-access-wtphs" (OuterVolumeSpecName: "kube-api-access-wtphs") pod "0558e524-d852-49bc-ae78-a02b954f27b3" (UID: "0558e524-d852-49bc-ae78-a02b954f27b3"). InnerVolumeSpecName "kube-api-access-wtphs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:10:02.179187 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.179168    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-dshm" (OuterVolumeSpecName: "dshm") pod "0558e524-d852-49bc-ae78-a02b954f27b3" (UID: "0558e524-d852-49bc-ae78-a02b954f27b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:10:02.277503 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.277465    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0558e524-d852-49bc-ae78-a02b954f27b3-tls-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:02.277503 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.277494    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-tmp-dir\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:02.277503 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.277506    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtphs\" (UniqueName: \"kubernetes.io/projected/0558e524-d852-49bc-ae78-a02b954f27b3-kube-api-access-wtphs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:02.277746 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.277517    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0558e524-d852-49bc-ae78-a02b954f27b3-dshm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:02.608202 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.608166    2550 generic.go:356] "Generic (PLEG): container finished" podID="0558e524-d852-49bc-ae78-a02b954f27b3" containerID="a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5" exitCode=0
Jul 20 20:10:02.608391 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.608211    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"
Jul 20 20:10:02.608391 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.608244    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn" event={"ID":"0558e524-d852-49bc-ae78-a02b954f27b3","Type":"ContainerDied","Data":"a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5"}
Jul 20 20:10:02.608391 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.608307    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn" event={"ID":"0558e524-d852-49bc-ae78-a02b954f27b3","Type":"ContainerDied","Data":"a23740283327f1b2cf840e587894a2c5930db95df2bb4055d302aa127a009190"}
Jul 20 20:10:02.608391 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.608326    2550 scope.go:117] "RemoveContainer" containerID="a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5"
Jul 20 20:10:02.618444 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.618403    2550 scope.go:117] "RemoveContainer" containerID="a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5"
Jul 20 20:10:02.618940 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:10:02.618910    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5\": container with ID starting with a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5 not found: ID does not exist" containerID="a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5"
Jul 20 20:10:02.619071 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.618945    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5"} err="failed to get container status \"a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5\": rpc error: code = NotFound desc = could not find container \"a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5\": container with ID starting with a911991a4dcf627aa9cfc6dd5c398c9ad25f4de51356e5a1d5e68132565b62f5 not found: ID does not exist"
Jul 20 20:10:02.627061 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.627039    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"]
Jul 20 20:10:02.633079 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.633060    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/sub-test-model-397b55-kserve-6f57f4c75c-qdfrn"]
Jul 20 20:10:02.645370 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.645342    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr" podUID="12070920-2ff6-472a-9bc6-006232430ab5" containerName="main" containerID="cri-o://e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195" gracePeriod=45
Jul 20 20:10:02.883185 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.883160    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:10:02.982972 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.982937    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-tmp-dir\") pod \"12070920-2ff6-472a-9bc6-006232430ab5\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") "
Jul 20 20:10:02.982972 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.982976    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12070920-2ff6-472a-9bc6-006232430ab5-tls-certs\") pod \"12070920-2ff6-472a-9bc6-006232430ab5\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") "
Jul 20 20:10:02.983217 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.982997    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-dshm\") pod \"12070920-2ff6-472a-9bc6-006232430ab5\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") "
Jul 20 20:10:02.983217 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.983024    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-model-cache\") pod \"12070920-2ff6-472a-9bc6-006232430ab5\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") "
Jul 20 20:10:02.983217 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.983055    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vgjt\" (UniqueName: \"kubernetes.io/projected/12070920-2ff6-472a-9bc6-006232430ab5-kube-api-access-6vgjt\") pod \"12070920-2ff6-472a-9bc6-006232430ab5\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") "
Jul 20 20:10:02.983217 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.983094    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-home\") pod \"12070920-2ff6-472a-9bc6-006232430ab5\" (UID: \"12070920-2ff6-472a-9bc6-006232430ab5\") "
Jul 20 20:10:02.983453 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.983248    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12070920-2ff6-472a-9bc6-006232430ab5" (UID: "12070920-2ff6-472a-9bc6-006232430ab5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:10:02.983453 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.983391    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-model-cache" (OuterVolumeSpecName: "model-cache") pod "12070920-2ff6-472a-9bc6-006232430ab5" (UID: "12070920-2ff6-472a-9bc6-006232430ab5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:10:02.983453 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.983400    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-tmp-dir\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:02.983585 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.983458    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-home" (OuterVolumeSpecName: "home") pod "12070920-2ff6-472a-9bc6-006232430ab5" (UID: "12070920-2ff6-472a-9bc6-006232430ab5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:10:02.985192 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.985170    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12070920-2ff6-472a-9bc6-006232430ab5-kube-api-access-6vgjt" (OuterVolumeSpecName: "kube-api-access-6vgjt") pod "12070920-2ff6-472a-9bc6-006232430ab5" (UID: "12070920-2ff6-472a-9bc6-006232430ab5"). InnerVolumeSpecName "kube-api-access-6vgjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:10:02.985291 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.985212    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12070920-2ff6-472a-9bc6-006232430ab5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12070920-2ff6-472a-9bc6-006232430ab5" (UID: "12070920-2ff6-472a-9bc6-006232430ab5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:10:02.985291 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:02.985216    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-dshm" (OuterVolumeSpecName: "dshm") pod "12070920-2ff6-472a-9bc6-006232430ab5" (UID: "12070920-2ff6-472a-9bc6-006232430ab5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:10:03.083901 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.083809    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-home\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:03.083901 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.083842    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12070920-2ff6-472a-9bc6-006232430ab5-tls-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:03.083901 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.083852    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-dshm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:03.083901 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.083861    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12070920-2ff6-472a-9bc6-006232430ab5-model-cache\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:03.083901 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.083871    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vgjt\" (UniqueName: \"kubernetes.io/projected/12070920-2ff6-472a-9bc6-006232430ab5-kube-api-access-6vgjt\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:10:03.613552 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.613516    2550 generic.go:356] "Generic (PLEG): container finished" podID="12070920-2ff6-472a-9bc6-006232430ab5" containerID="e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195" exitCode=0
Jul 20 20:10:03.613771 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.613560    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"
Jul 20 20:10:03.613771 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.613606    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr" event={"ID":"12070920-2ff6-472a-9bc6-006232430ab5","Type":"ContainerDied","Data":"e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195"}
Jul 20 20:10:03.613771 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.613650    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr" event={"ID":"12070920-2ff6-472a-9bc6-006232430ab5","Type":"ContainerDied","Data":"c2e3a7bdb17d2ed6b592ee7498cc35f1b8bd1768385800e4c55d2b928361f145"}
Jul 20 20:10:03.613771 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.613669    2550 scope.go:117] "RemoveContainer" containerID="e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195"
Jul 20 20:10:03.622435 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.622415    2550 scope.go:117] "RemoveContainer" containerID="e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195"
Jul 20 20:10:03.622689 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:10:03.622668    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195\": container with ID starting with e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195 not found: ID does not exist" containerID="e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195"
Jul 20 20:10:03.622755 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.622699    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195"} err="failed to get container status \"e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195\": rpc error: code = NotFound desc = could not find container \"e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195\": container with ID starting with e2373be908f2b93580d0ab2a8cc4cac8f2b4c1c28b9d8760e661647b8c77b195 not found: ID does not exist"
Jul 20 20:10:03.635987 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.635958    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"]
Jul 20 20:10:03.641864 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:03.641838    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/sub-test-model-6fb9c1-kserve-5dd79fff45-pxbkr"]
Jul 20 20:10:04.450760 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:04.450726    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0558e524-d852-49bc-ae78-a02b954f27b3" path="/var/lib/kubelet/pods/0558e524-d852-49bc-ae78-a02b954f27b3/volumes"
Jul 20 20:10:04.451142 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:04.451106    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12070920-2ff6-472a-9bc6-006232430ab5" path="/var/lib/kubelet/pods/12070920-2ff6-472a-9bc6-006232430ab5/volumes"
Jul 20 20:10:14.201243 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.201203    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"]
Jul 20 20:10:14.201733 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.201708    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12070920-2ff6-472a-9bc6-006232430ab5" containerName="main"
Jul 20 20:10:14.201784 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.201738    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="12070920-2ff6-472a-9bc6-006232430ab5" containerName="main"
Jul 20 20:10:14.201784 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.201780    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0558e524-d852-49bc-ae78-a02b954f27b3" containerName="main"
Jul 20 20:10:14.201862 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.201789    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0558e524-d852-49bc-ae78-a02b954f27b3" containerName="main"
Jul 20 20:10:14.201902 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.201866    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0558e524-d852-49bc-ae78-a02b954f27b3" containerName="main"
Jul 20 20:10:14.201902 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.201881    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="12070920-2ff6-472a-9bc6-006232430ab5" containerName="main"
Jul 20 20:10:14.204869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.204848    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.207314 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.207289    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"rate-test-model-6fb9c1-kserve-self-signed-certs\""
Jul 20 20:10:14.208092 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.208071    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"default-dockercfg-w2vrt\""
Jul 20 20:10:14.208208 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.208101    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"kube-root-ca.crt\""
Jul 20 20:10:14.208208 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.208101    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6fb9c1\"/\"openshift-service-ca.crt\""
Jul 20 20:10:14.217040 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.217007    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"]
Jul 20 20:10:14.274868 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.274831    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-model-cache\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.274868 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.274866    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tls-certs\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.275098 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.274900    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcq82\" (UniqueName: \"kubernetes.io/projected/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-kube-api-access-rcq82\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.275098 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.274995    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tmp-dir\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.275098 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.275027    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-dshm\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.275098 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.275065    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-home\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.376235 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.376200    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-home\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.376428 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.376273    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-model-cache\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.376428 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.376296    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tls-certs\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.376428 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.376329    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcq82\" (UniqueName: \"kubernetes.io/projected/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-kube-api-access-rcq82\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.376428 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.376360    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tmp-dir\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.376428 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.376387    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-dshm\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.376721 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.376650    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-model-cache\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.376721 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.376709    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-home\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.376804 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.376739    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tmp-dir\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.378609 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.378587    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-dshm\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.378912 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.378895    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tls-certs\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.384844 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.384813    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcq82\" (UniqueName: \"kubernetes.io/projected/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-kube-api-access-rcq82\") pod \"rate-test-model-6fb9c1-kserve-555d49556c-47s8z\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") " pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.516607 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.516527    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:14.649050 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.649021    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"]
Jul 20 20:10:14.651767 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:10:14.651741    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d4f4368_479e_4527_b0ee_f2cb93d7e26e.slice/crio-81799b1970aab48f7d0ca726c945dd8f67543c895485e5fc051b5765473f3b36 WatchSource:0}: Error finding container 81799b1970aab48f7d0ca726c945dd8f67543c895485e5fc051b5765473f3b36: Status 404 returned error can't find the container with id 81799b1970aab48f7d0ca726c945dd8f67543c895485e5fc051b5765473f3b36
Jul 20 20:10:14.666296 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:14.666223    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z" event={"ID":"5d4f4368-479e-4527-b0ee-f2cb93d7e26e","Type":"ContainerStarted","Data":"81799b1970aab48f7d0ca726c945dd8f67543c895485e5fc051b5765473f3b36"}
Jul 20 20:10:15.672925 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:15.672880    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z" event={"ID":"5d4f4368-479e-4527-b0ee-f2cb93d7e26e","Type":"ContainerStarted","Data":"933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8"}
Jul 20 20:10:15.690699 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:15.690648    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z" podStartSLOduration=1.69062993 podStartE2EDuration="1.69062993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:10:15.689790319 +0000 UTC m=+2867.775741317" watchObservedRunningTime="2026-07-20 20:10:15.69062993 +0000 UTC m=+2867.776580928"
Jul 20 20:10:24.516868 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:24.516778    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:24.517403 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:24.517205    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:24.534244 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:24.534215    2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:24.725311 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:24.725280    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:10:28.743562 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:10:28.743529    2550 scope.go:117] "RemoveContainer" containerID="bc283fbc0f41f38debb0186a56beb4e37503db86ff4f9894b8f5a92b8ada84c8"
Jul 20 20:11:01.042516 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:01.042475    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"]
Jul 20 20:11:01.834348 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:01.834313    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"]
Jul 20 20:11:16.062272 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.062220    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl" podUID="16aba4c9-0e95-44fb-8029-833e4051d251" containerName="main" containerID="cri-o://4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd" gracePeriod=45
Jul 20 20:11:16.309884 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.309858    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:11:16.375870 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.375840    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-model-cache\") pod \"16aba4c9-0e95-44fb-8029-833e4051d251\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") "
Jul 20 20:11:16.376040 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.375881    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdl2x\" (UniqueName: \"kubernetes.io/projected/16aba4c9-0e95-44fb-8029-833e4051d251-kube-api-access-qdl2x\") pod \"16aba4c9-0e95-44fb-8029-833e4051d251\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") "
Jul 20 20:11:16.376040 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.375901    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-home\") pod \"16aba4c9-0e95-44fb-8029-833e4051d251\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") "
Jul 20 20:11:16.376040 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.375936    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-dshm\") pod \"16aba4c9-0e95-44fb-8029-833e4051d251\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") "
Jul 20 20:11:16.376040 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.375957    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16aba4c9-0e95-44fb-8029-833e4051d251-tls-certs\") pod \"16aba4c9-0e95-44fb-8029-833e4051d251\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") "
Jul 20 20:11:16.376040 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.375976    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-tmp-dir\") pod \"16aba4c9-0e95-44fb-8029-833e4051d251\" (UID: \"16aba4c9-0e95-44fb-8029-833e4051d251\") "
Jul 20 20:11:16.376359 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.376134    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-model-cache" (OuterVolumeSpecName: "model-cache") pod "16aba4c9-0e95-44fb-8029-833e4051d251" (UID: "16aba4c9-0e95-44fb-8029-833e4051d251"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:16.376359 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.376222    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-home" (OuterVolumeSpecName: "home") pod "16aba4c9-0e95-44fb-8029-833e4051d251" (UID: "16aba4c9-0e95-44fb-8029-833e4051d251"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:16.376359 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.376347    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-model-cache\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:16.376359 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.376363    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-home\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:16.376524 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.376363    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16aba4c9-0e95-44fb-8029-833e4051d251" (UID: "16aba4c9-0e95-44fb-8029-833e4051d251"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:16.378150 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.378119    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16aba4c9-0e95-44fb-8029-833e4051d251-kube-api-access-qdl2x" (OuterVolumeSpecName: "kube-api-access-qdl2x") pod "16aba4c9-0e95-44fb-8029-833e4051d251" (UID: "16aba4c9-0e95-44fb-8029-833e4051d251"). InnerVolumeSpecName "kube-api-access-qdl2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:11:16.378150 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.378137    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16aba4c9-0e95-44fb-8029-833e4051d251-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16aba4c9-0e95-44fb-8029-833e4051d251" (UID: "16aba4c9-0e95-44fb-8029-833e4051d251"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:11:16.378150 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.378119    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-dshm" (OuterVolumeSpecName: "dshm") pod "16aba4c9-0e95-44fb-8029-833e4051d251" (UID: "16aba4c9-0e95-44fb-8029-833e4051d251"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:16.476730 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.476696    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdl2x\" (UniqueName: \"kubernetes.io/projected/16aba4c9-0e95-44fb-8029-833e4051d251-kube-api-access-qdl2x\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:16.476730 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.476727    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-dshm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:16.476918 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.476741    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16aba4c9-0e95-44fb-8029-833e4051d251-tls-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:16.476918 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.476753    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16aba4c9-0e95-44fb-8029-833e4051d251-tmp-dir\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:16.854789 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.854671    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z" podUID="5d4f4368-479e-4527-b0ee-f2cb93d7e26e" containerName="main" containerID="cri-o://933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8" gracePeriod=45
Jul 20 20:11:16.953515 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.953478    2550 generic.go:356] "Generic (PLEG): container finished" podID="16aba4c9-0e95-44fb-8029-833e4051d251" containerID="4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd" exitCode=0
Jul 20 20:11:16.953696 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.953534    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"
Jul 20 20:11:16.953696 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.953564    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl" event={"ID":"16aba4c9-0e95-44fb-8029-833e4051d251","Type":"ContainerDied","Data":"4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd"}
Jul 20 20:11:16.953696 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.953607    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl" event={"ID":"16aba4c9-0e95-44fb-8029-833e4051d251","Type":"ContainerDied","Data":"7d6ec8a9ee241d4f9ed9c99132a3cdf6125c4c239f1c70b886b90ed70446f00e"}
Jul 20 20:11:16.953696 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.953631    2550 scope.go:117] "RemoveContainer" containerID="4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd"
Jul 20 20:11:16.964924 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.964906    2550 scope.go:117] "RemoveContainer" containerID="4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd"
Jul 20 20:11:16.965209 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:11:16.965188    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd\": container with ID starting with 4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd not found: ID does not exist" containerID="4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd"
Jul 20 20:11:16.965325 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.965219    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd"} err="failed to get container status \"4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd\": rpc error: code = NotFound desc = could not find container \"4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd\": container with ID starting with 4a05ae4280070793da3ccf49d8780b5a8b0ae9d771c639c74354b3653abcbbcd not found: ID does not exist"
Jul 20 20:11:16.979108 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.979082    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"]
Jul 20 20:11:16.987511 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:16.987485    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-397b55/rate-test-model-397b55-kserve-5f87487d95-rwjvl"]
Jul 20 20:11:17.105978 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.105920    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:11:17.186745 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.186708    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-home\") pod \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") "
Jul 20 20:11:17.186929 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.186753    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tmp-dir\") pod \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") "
Jul 20 20:11:17.186929 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.186783    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-model-cache\") pod \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") "
Jul 20 20:11:17.186929 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.186818    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcq82\" (UniqueName: \"kubernetes.io/projected/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-kube-api-access-rcq82\") pod \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") "
Jul 20 20:11:17.186929 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.186833    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tls-certs\") pod \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") "
Jul 20 20:11:17.186929 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.186856    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-dshm\") pod \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\" (UID: \"5d4f4368-479e-4527-b0ee-f2cb93d7e26e\") "
Jul 20 20:11:17.187228 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.187025    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-home" (OuterVolumeSpecName: "home") pod "5d4f4368-479e-4527-b0ee-f2cb93d7e26e" (UID: "5d4f4368-479e-4527-b0ee-f2cb93d7e26e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:17.187228 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.187083    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d4f4368-479e-4527-b0ee-f2cb93d7e26e" (UID: "5d4f4368-479e-4527-b0ee-f2cb93d7e26e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:17.187228 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.187092    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d4f4368-479e-4527-b0ee-f2cb93d7e26e" (UID: "5d4f4368-479e-4527-b0ee-f2cb93d7e26e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:17.189069 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.189041    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-dshm" (OuterVolumeSpecName: "dshm") pod "5d4f4368-479e-4527-b0ee-f2cb93d7e26e" (UID: "5d4f4368-479e-4527-b0ee-f2cb93d7e26e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:17.189187 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.189069    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d4f4368-479e-4527-b0ee-f2cb93d7e26e" (UID: "5d4f4368-479e-4527-b0ee-f2cb93d7e26e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:11:17.189187 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.189087    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-kube-api-access-rcq82" (OuterVolumeSpecName: "kube-api-access-rcq82") pod "5d4f4368-479e-4527-b0ee-f2cb93d7e26e" (UID: "5d4f4368-479e-4527-b0ee-f2cb93d7e26e"). InnerVolumeSpecName "kube-api-access-rcq82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:11:17.287848 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.287812    2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-home\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:17.287848 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.287840    2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tmp-dir\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:17.287848 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.287850    2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-model-cache\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:17.287848 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.287859    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcq82\" (UniqueName: \"kubernetes.io/projected/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-kube-api-access-rcq82\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:17.288115 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.287868    2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-tls-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:17.288115 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.287876    2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d4f4368-479e-4527-b0ee-f2cb93d7e26e-dshm\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:17.960601 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.960567    2550 generic.go:356] "Generic (PLEG): container finished" podID="5d4f4368-479e-4527-b0ee-f2cb93d7e26e" containerID="933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8" exitCode=0
Jul 20 20:11:17.960784 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.960611    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z" event={"ID":"5d4f4368-479e-4527-b0ee-f2cb93d7e26e","Type":"ContainerDied","Data":"933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8"}
Jul 20 20:11:17.960784 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.960639    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z" event={"ID":"5d4f4368-479e-4527-b0ee-f2cb93d7e26e","Type":"ContainerDied","Data":"81799b1970aab48f7d0ca726c945dd8f67543c895485e5fc051b5765473f3b36"}
Jul 20 20:11:17.960784 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.960657    2550 scope.go:117] "RemoveContainer" containerID="933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8"
Jul 20 20:11:17.960784 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.960614    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"
Jul 20 20:11:17.969237 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.969214    2550 scope.go:117] "RemoveContainer" containerID="933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8"
Jul 20 20:11:17.969518 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:11:17.969496    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8\": container with ID starting with 933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8 not found: ID does not exist" containerID="933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8"
Jul 20 20:11:17.969567 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.969528    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8"} err="failed to get container status \"933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8\": rpc error: code = NotFound desc = could not find container \"933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8\": container with ID starting with 933f6bee7f88b04ba3297dd30a7cdda6984e8ca98f1bdfcbb38e6a250c130dc8 not found: ID does not exist"
Jul 20 20:11:17.990465 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:17.990437    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"]
Jul 20 20:11:18.004677 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:18.004650    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6fb9c1/rate-test-model-6fb9c1-kserve-555d49556c-47s8z"]
Jul 20 20:11:18.451001 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:18.450955    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16aba4c9-0e95-44fb-8029-833e4051d251" path="/var/lib/kubelet/pods/16aba4c9-0e95-44fb-8029-833e4051d251/volumes"
Jul 20 20:11:18.451551 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:18.451530    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d4f4368-479e-4527-b0ee-f2cb93d7e26e" path="/var/lib/kubelet/pods/5d4f4368-479e-4527-b0ee-f2cb93d7e26e/volumes"
Jul 20 20:11:31.636689 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:31.636654    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"]
Jul 20 20:11:31.637201 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:31.636918    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l" podUID="cd81587d-00ac-4531-9406-855f481c0272" containerName="maas-api" containerID="cri-o://9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2" gracePeriod=30
Jul 20 20:11:31.890252 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:31.890183    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:11:32.008612 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.008570    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd81587d-00ac-4531-9406-855f481c0272-maas-api-tls\") pod \"cd81587d-00ac-4531-9406-855f481c0272\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") "
Jul 20 20:11:32.008866 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.008630    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd81587d-00ac-4531-9406-855f481c0272-openshift-service-ca\") pod \"cd81587d-00ac-4531-9406-855f481c0272\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") "
Jul 20 20:11:32.008866 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.008662    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hffkf\" (UniqueName: \"kubernetes.io/projected/cd81587d-00ac-4531-9406-855f481c0272-kube-api-access-hffkf\") pod \"cd81587d-00ac-4531-9406-855f481c0272\" (UID: \"cd81587d-00ac-4531-9406-855f481c0272\") "
Jul 20 20:11:32.009041 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.009016    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd81587d-00ac-4531-9406-855f481c0272-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd81587d-00ac-4531-9406-855f481c0272" (UID: "cd81587d-00ac-4531-9406-855f481c0272"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 20:11:32.010832 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.010808    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd81587d-00ac-4531-9406-855f481c0272-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd81587d-00ac-4531-9406-855f481c0272" (UID: "cd81587d-00ac-4531-9406-855f481c0272"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:11:32.010911 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.010841    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd81587d-00ac-4531-9406-855f481c0272-kube-api-access-hffkf" (OuterVolumeSpecName: "kube-api-access-hffkf") pod "cd81587d-00ac-4531-9406-855f481c0272" (UID: "cd81587d-00ac-4531-9406-855f481c0272"). InnerVolumeSpecName "kube-api-access-hffkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:11:32.023210 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.023182    2550 generic.go:356] "Generic (PLEG): container finished" podID="cd81587d-00ac-4531-9406-855f481c0272" containerID="9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2" exitCode=0
Jul 20 20:11:32.023346 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.023230    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"
Jul 20 20:11:32.023346 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.023283    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l" event={"ID":"cd81587d-00ac-4531-9406-855f481c0272","Type":"ContainerDied","Data":"9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2"}
Jul 20 20:11:32.023346 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.023336    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l" event={"ID":"cd81587d-00ac-4531-9406-855f481c0272","Type":"ContainerDied","Data":"dbd367d849298c0a69d4c2a8d22283845658e0cfcb92bb71d789641794dd984e"}
Jul 20 20:11:32.023476 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.023359    2550 scope.go:117] "RemoveContainer" containerID="9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2"
Jul 20 20:11:32.032558 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.032542    2550 scope.go:117] "RemoveContainer" containerID="9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2"
Jul 20 20:11:32.032811 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:11:32.032791    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2\": container with ID starting with 9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2 not found: ID does not exist" containerID="9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2"
Jul 20 20:11:32.032870 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.032821    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2"} err="failed to get container status \"9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2\": rpc error: code = NotFound desc = could not find container \"9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2\": container with ID starting with 9c0917af121b31ba2b47af7e2e28bf442ed6d191355c7f0da6880fae7a06fed2 not found: ID does not exist"
Jul 20 20:11:32.044830 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.044799    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"]
Jul 20 20:11:32.048965 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.048936    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-397b55-6bb97d9d57-g448l"]
Jul 20 20:11:32.110087 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.110048    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hffkf\" (UniqueName: \"kubernetes.io/projected/cd81587d-00ac-4531-9406-855f481c0272-kube-api-access-hffkf\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:32.110087 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.110081    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd81587d-00ac-4531-9406-855f481c0272-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:32.110087 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.110095    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd81587d-00ac-4531-9406-855f481c0272-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:32.450726 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:32.450689    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd81587d-00ac-4531-9406-855f481c0272" path="/var/lib/kubelet/pods/cd81587d-00ac-4531-9406-855f481c0272/volumes"
Jul 20 20:11:42.877069 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:42.877034    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"]
Jul 20 20:11:42.877519 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:42.877286    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy" containerID="cri-o://b5bfbbed26b8627a81995cce4461925683adda61e9bd57bfabd8ecec19eeb46a" gracePeriod=30
Jul 20 20:11:48.094408 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.094371    2550 generic.go:356] "Generic (PLEG): container finished" podID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerID="b5bfbbed26b8627a81995cce4461925683adda61e9bd57bfabd8ecec19eeb46a" exitCode=0
Jul 20 20:11:48.094907 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.094439    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" event={"ID":"f0e78969-7bc0-470a-ad43-8860b4c312eb","Type":"ContainerDied","Data":"b5bfbbed26b8627a81995cce4461925683adda61e9bd57bfabd8ecec19eeb46a"}
Jul 20 20:11:48.094907 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.094497    2550 scope.go:117] "RemoveContainer" containerID="126f34b335ae841097b32ee0ab36c7acdde70d22ac02fe6b32e20f772e51e1de"
Jul 20 20:11:48.130417 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.130389    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:11:48.245234 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245132    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-socket\") pod \"f0e78969-7bc0-470a-ad43-8860b4c312eb\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") "
Jul 20 20:11:48.245234 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245190    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-envoy\") pod \"f0e78969-7bc0-470a-ad43-8860b4c312eb\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") "
Jul 20 20:11:48.245234 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245227    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-credential-socket\") pod \"f0e78969-7bc0-470a-ad43-8860b4c312eb\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") "
Jul 20 20:11:48.245541 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245248    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-data\") pod \"f0e78969-7bc0-470a-ad43-8860b4c312eb\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") "
Jul 20 20:11:48.245541 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245290    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-token\") pod \"f0e78969-7bc0-470a-ad43-8860b4c312eb\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") "
Jul 20 20:11:48.245541 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245327    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-certs\") pod \"f0e78969-7bc0-470a-ad43-8860b4c312eb\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") "
Jul 20 20:11:48.245541 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245355    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-podinfo\") pod \"f0e78969-7bc0-470a-ad43-8860b4c312eb\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") "
Jul 20 20:11:48.245541 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245403    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55d74\" (UniqueName: \"kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-kube-api-access-55d74\") pod \"f0e78969-7bc0-470a-ad43-8860b4c312eb\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") "
Jul 20 20:11:48.245541 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245443    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0e78969-7bc0-470a-ad43-8860b4c312eb-istiod-ca-cert\") pod \"f0e78969-7bc0-470a-ad43-8860b4c312eb\" (UID: \"f0e78969-7bc0-470a-ad43-8860b4c312eb\") "
Jul 20 20:11:48.245541 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245480    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0e78969-7bc0-470a-ad43-8860b4c312eb" (UID: "f0e78969-7bc0-470a-ad43-8860b4c312eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:48.245914 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245574    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0e78969-7bc0-470a-ad43-8860b4c312eb" (UID: "f0e78969-7bc0-470a-ad43-8860b4c312eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:48.245914 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245709    2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:48.245914 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245730    2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-credential-socket\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:48.245914 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245739    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0e78969-7bc0-470a-ad43-8860b4c312eb" (UID: "f0e78969-7bc0-470a-ad43-8860b4c312eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:48.246105 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.245917    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e78969-7bc0-470a-ad43-8860b4c312eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0e78969-7bc0-470a-ad43-8860b4c312eb" (UID: "f0e78969-7bc0-470a-ad43-8860b4c312eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 20:11:48.246408 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.246385    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0e78969-7bc0-470a-ad43-8860b4c312eb" (UID: "f0e78969-7bc0-470a-ad43-8860b4c312eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:48.247612 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.247591    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0e78969-7bc0-470a-ad43-8860b4c312eb" (UID: "f0e78969-7bc0-470a-ad43-8860b4c312eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:11:48.247736 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.247632    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-kube-api-access-55d74" (OuterVolumeSpecName: "kube-api-access-55d74") pod "f0e78969-7bc0-470a-ad43-8860b4c312eb" (UID: "f0e78969-7bc0-470a-ad43-8860b4c312eb"). InnerVolumeSpecName "kube-api-access-55d74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:11:48.247888 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.247868    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0e78969-7bc0-470a-ad43-8860b4c312eb" (UID: "f0e78969-7bc0-470a-ad43-8860b4c312eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 20 20:11:48.247947 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.247903    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0e78969-7bc0-470a-ad43-8860b4c312eb" (UID: "f0e78969-7bc0-470a-ad43-8860b4c312eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 20 20:11:48.346307 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.346242    2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0e78969-7bc0-470a-ad43-8860b4c312eb-istiod-ca-cert\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:48.346307 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.346298    2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-envoy\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:48.346307 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.346307    2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-data\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:48.346307 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.346315    2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-token\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:48.346575 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.346323    2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e78969-7bc0-470a-ad43-8860b4c312eb-workload-certs\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:48.346575 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.346332    2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0e78969-7bc0-470a-ad43-8860b4c312eb-istio-podinfo\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:48.346575 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:48.346341    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55d74\" (UniqueName: \"kubernetes.io/projected/f0e78969-7bc0-470a-ad43-8860b4c312eb-kube-api-access-55d74\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:49.099150 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:49.099115    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"
Jul 20 20:11:49.099585 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:49.099164    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67" event={"ID":"f0e78969-7bc0-470a-ad43-8860b4c312eb","Type":"ContainerDied","Data":"56b788e86cbf074f798905d1c11f2e89ef35b29f85bc8345aa923835d24b5219"}
Jul 20 20:11:49.099585 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:49.099218    2550 scope.go:117] "RemoveContainer" containerID="b5bfbbed26b8627a81995cce4461925683adda61e9bd57bfabd8ecec19eeb46a"
Jul 20 20:11:49.119472 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:49.119448    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"]
Jul 20 20:11:49.123728 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:49.123703    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-397b55-openshift-default-cb5764759-5qj67"]
Jul 20 20:11:50.450670 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:50.450636    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" path="/var/lib/kubelet/pods/f0e78969-7bc0-470a-ad43-8860b4c312eb/volumes"
Jul 20 20:11:53.770056 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:53.770019    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"]
Jul 20 20:11:53.770488 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:53.770326    2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf" podUID="9b7ac406-3972-4bca-a827-2ebab0cc57a3" containerName="maas-api" containerID="cri-o://66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e" gracePeriod=30
Jul 20 20:11:54.015536 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.015507    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:11:54.093877 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.093782    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7ac406-3972-4bca-a827-2ebab0cc57a3-openshift-service-ca\") pod \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") "
Jul 20 20:11:54.093877 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.093834    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7ac406-3972-4bca-a827-2ebab0cc57a3-maas-api-tls\") pod \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") "
Jul 20 20:11:54.093877 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.093863    2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbp2p\" (UniqueName: \"kubernetes.io/projected/9b7ac406-3972-4bca-a827-2ebab0cc57a3-kube-api-access-fbp2p\") pod \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\" (UID: \"9b7ac406-3972-4bca-a827-2ebab0cc57a3\") "
Jul 20 20:11:54.094183 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.094156    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7ac406-3972-4bca-a827-2ebab0cc57a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b7ac406-3972-4bca-a827-2ebab0cc57a3" (UID: "9b7ac406-3972-4bca-a827-2ebab0cc57a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 20 20:11:54.095968 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.095943    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7ac406-3972-4bca-a827-2ebab0cc57a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b7ac406-3972-4bca-a827-2ebab0cc57a3" (UID: "9b7ac406-3972-4bca-a827-2ebab0cc57a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 20 20:11:54.096072 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.095999    2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7ac406-3972-4bca-a827-2ebab0cc57a3-kube-api-access-fbp2p" (OuterVolumeSpecName: "kube-api-access-fbp2p") pod "9b7ac406-3972-4bca-a827-2ebab0cc57a3" (UID: "9b7ac406-3972-4bca-a827-2ebab0cc57a3"). InnerVolumeSpecName "kube-api-access-fbp2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 20 20:11:54.127643 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.127609    2550 generic.go:356] "Generic (PLEG): container finished" podID="9b7ac406-3972-4bca-a827-2ebab0cc57a3" containerID="66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e" exitCode=0
Jul 20 20:11:54.127810 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.127658    2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"
Jul 20 20:11:54.127810 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.127699    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf" event={"ID":"9b7ac406-3972-4bca-a827-2ebab0cc57a3","Type":"ContainerDied","Data":"66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e"}
Jul 20 20:11:54.127810 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.127750    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf" event={"ID":"9b7ac406-3972-4bca-a827-2ebab0cc57a3","Type":"ContainerDied","Data":"e4c942cfd215bd326ff3a6b3133f3ca6483a1d68383e6981b4dd26e8ba0bef51"}
Jul 20 20:11:54.127810 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.127772    2550 scope.go:117] "RemoveContainer" containerID="66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e"
Jul 20 20:11:54.136400 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.136384    2550 scope.go:117] "RemoveContainer" containerID="66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e"
Jul 20 20:11:54.136665 ip-10-0-138-227 kubenswrapper[2550]: E0720 20:11:54.136646    2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e\": container with ID starting with 66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e not found: ID does not exist" containerID="66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e"
Jul 20 20:11:54.136738 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.136673    2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e"} err="failed to get container status \"66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e\": rpc error: code = NotFound desc = could not find container \"66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e\": container with ID starting with 66da101d2e5437fc03c4528e4b4d3bdb1e754c2fa2895fb86cca589abd60978e not found: ID does not exist"
Jul 20 20:11:54.147683 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.147658    2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"]
Jul 20 20:11:54.153964 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.153941    2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-6fb9c1-69c56f48cb-rlsnf"]
Jul 20 20:11:54.194576 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.194540    2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b7ac406-3972-4bca-a827-2ebab0cc57a3-openshift-service-ca\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:54.194576 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.194572    2550 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b7ac406-3972-4bca-a827-2ebab0cc57a3-maas-api-tls\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:54.194576 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.194581    2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbp2p\" (UniqueName: \"kubernetes.io/projected/9b7ac406-3972-4bca-a827-2ebab0cc57a3-kube-api-access-fbp2p\") on node \"ip-10-0-138-227.ec2.internal\" DevicePath \"\""
Jul 20 20:11:54.450101 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:11:54.450066    2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b7ac406-3972-4bca-a827-2ebab0cc57a3" path="/var/lib/kubelet/pods/9b7ac406-3972-4bca-a827-2ebab0cc57a3/volumes"
Jul 20 20:12:05.890701 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:05.890661    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85756d6b4-c2zbv_d82fd48a-0bd3-4367-a994-6079e3cf5897/authorino/0.log"
Jul 20 20:12:11.696045 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:11.696011    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75c5bf7b68-9db78_6de81c58-514d-4d01-9d9b-77e54012c519/manager/0.log"
Jul 20 20:12:11.920521 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:11.920477    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n7b9c_13b6218d-ee95-4de8-8171-26749fc7da1d/manager/0.log"
Jul 20 20:12:12.037764 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:12.037680    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr_76d56700-1f80-437b-9565-9a3d46e7f3d9/copy-manifests/0.log"
Jul 20 20:12:12.044740 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:12.044711    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr_76d56700-1f80-437b-9565-9a3d46e7f3d9/manager/0.log"
Jul 20 20:12:12.676358 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:12.676331    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-9f859c8f8-vf9lt_524b1c09-1584-48ee-b2aa-bbd0bd65102c/maas-api/0.log"
Jul 20 20:12:14.375235 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:14.375206    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85756d6b4-c2zbv_d82fd48a-0bd3-4367-a994-6079e3cf5897/authorino/0.log"
Jul 20 20:12:14.731183 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:14.731105    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9nzjq_53d4d1e3-1196-4995-a421-44c48295b555/kuadrant-console-plugin/0.log"
Jul 20 20:12:15.967943 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:15.967912    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7578c486d7-vrkgc_45994ef8-30f2-48c1-8707-44b700e57a2b/kube-auth-proxy/0.log"
Jul 20 20:12:16.327645 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:16.327569    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-4chbp_e19a7680-c1b3-49e7-8e02-e3d48ecba89d/payload-processing/0.log"
Jul 20 20:12:17.018381 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:17.018347    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-wrxnh_198f17cb-6252-420a-a634-c2f4419ce9e9/main/0.log"
Jul 20 20:12:23.303761 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.303724    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87kz5/must-gather-7v2s8"]
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304019    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b7ac406-3972-4bca-a827-2ebab0cc57a3" containerName="maas-api"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304030    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b7ac406-3972-4bca-a827-2ebab0cc57a3" containerName="maas-api"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304038    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd81587d-00ac-4531-9406-855f481c0272" containerName="maas-api"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304043    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd81587d-00ac-4531-9406-855f481c0272" containerName="maas-api"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304051    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16aba4c9-0e95-44fb-8029-833e4051d251" containerName="main"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304057    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="16aba4c9-0e95-44fb-8029-833e4051d251" containerName="main"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304068    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304074    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304086    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304091    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304098    2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d4f4368-479e-4527-b0ee-f2cb93d7e26e" containerName="main"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304104    2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d4f4368-479e-4527-b0ee-f2cb93d7e26e" containerName="main"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304148    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304157    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b7ac406-3972-4bca-a827-2ebab0cc57a3" containerName="maas-api"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304165    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd81587d-00ac-4531-9406-855f481c0272" containerName="maas-api"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304173    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="16aba4c9-0e95-44fb-8029-833e4051d251" containerName="main"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304179    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d4f4368-479e-4527-b0ee-f2cb93d7e26e" containerName="main"
Jul 20 20:12:23.304289 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.304287    2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e78969-7bc0-470a-ad43-8860b4c312eb" containerName="istio-proxy"
Jul 20 20:12:23.307296 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.307250    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87kz5/must-gather-7v2s8"
Jul 20 20:12:23.309700 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.309676    2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-87kz5\"/\"default-dockercfg-n49t4\""
Jul 20 20:12:23.309874 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.309703    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87kz5\"/\"kube-root-ca.crt\""
Jul 20 20:12:23.310383 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.310365    2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87kz5\"/\"openshift-service-ca.crt\""
Jul 20 20:12:23.319083 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.319057    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87kz5/must-gather-7v2s8"]
Jul 20 20:12:23.343557 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.343525    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzv62\" (UniqueName: \"kubernetes.io/projected/0d4a3e53-8b38-4f36-9bb1-32926a2c254f-kube-api-access-dzv62\") pod \"must-gather-7v2s8\" (UID: \"0d4a3e53-8b38-4f36-9bb1-32926a2c254f\") " pod="openshift-must-gather-87kz5/must-gather-7v2s8"
Jul 20 20:12:23.343557 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.343561    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0d4a3e53-8b38-4f36-9bb1-32926a2c254f-must-gather-output\") pod \"must-gather-7v2s8\" (UID: \"0d4a3e53-8b38-4f36-9bb1-32926a2c254f\") " pod="openshift-must-gather-87kz5/must-gather-7v2s8"
Jul 20 20:12:23.444530 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.444491    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzv62\" (UniqueName: \"kubernetes.io/projected/0d4a3e53-8b38-4f36-9bb1-32926a2c254f-kube-api-access-dzv62\") pod \"must-gather-7v2s8\" (UID: \"0d4a3e53-8b38-4f36-9bb1-32926a2c254f\") " pod="openshift-must-gather-87kz5/must-gather-7v2s8"
Jul 20 20:12:23.444530 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.444536    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0d4a3e53-8b38-4f36-9bb1-32926a2c254f-must-gather-output\") pod \"must-gather-7v2s8\" (UID: \"0d4a3e53-8b38-4f36-9bb1-32926a2c254f\") " pod="openshift-must-gather-87kz5/must-gather-7v2s8"
Jul 20 20:12:23.444869 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.444853    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0d4a3e53-8b38-4f36-9bb1-32926a2c254f-must-gather-output\") pod \"must-gather-7v2s8\" (UID: \"0d4a3e53-8b38-4f36-9bb1-32926a2c254f\") " pod="openshift-must-gather-87kz5/must-gather-7v2s8"
Jul 20 20:12:23.457640 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.457613    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzv62\" (UniqueName: \"kubernetes.io/projected/0d4a3e53-8b38-4f36-9bb1-32926a2c254f-kube-api-access-dzv62\") pod \"must-gather-7v2s8\" (UID: \"0d4a3e53-8b38-4f36-9bb1-32926a2c254f\") " pod="openshift-must-gather-87kz5/must-gather-7v2s8"
Jul 20 20:12:23.616147 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.616055    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87kz5/must-gather-7v2s8"
Jul 20 20:12:23.749514 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.749478    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87kz5/must-gather-7v2s8"]
Jul 20 20:12:23.752456 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:12:23.752432    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d4a3e53_8b38_4f36_9bb1_32926a2c254f.slice/crio-c7c523e047a49b191282e9a6bd82df13af90c5ffd3410663f62dd00c438db6a4 WatchSource:0}: Error finding container c7c523e047a49b191282e9a6bd82df13af90c5ffd3410663f62dd00c438db6a4: Status 404 returned error can't find the container with id c7c523e047a49b191282e9a6bd82df13af90c5ffd3410663f62dd00c438db6a4
Jul 20 20:12:23.754473 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:23.754453    2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 20 20:12:24.261509 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:24.261466    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87kz5/must-gather-7v2s8" event={"ID":"0d4a3e53-8b38-4f36-9bb1-32926a2c254f","Type":"ContainerStarted","Data":"c7c523e047a49b191282e9a6bd82df13af90c5ffd3410663f62dd00c438db6a4"}
Jul 20 20:12:25.268493 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:25.268453    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87kz5/must-gather-7v2s8" event={"ID":"0d4a3e53-8b38-4f36-9bb1-32926a2c254f","Type":"ContainerStarted","Data":"c81394cdae5378e2228da9dd072a89034b8ee099a07a0f1128d88d5b80f58b1b"}
Jul 20 20:12:25.268958 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:25.268506    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87kz5/must-gather-7v2s8" event={"ID":"0d4a3e53-8b38-4f36-9bb1-32926a2c254f","Type":"ContainerStarted","Data":"facf40518e9b21d1b09dad76f8d12f1de5661bb060fbd06bc004932a94bb2f20"}
Jul 20 20:12:25.283378 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:25.283327    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87kz5/must-gather-7v2s8" podStartSLOduration=1.614660749 podStartE2EDuration="2.283311011s" totalImagesPullingTime="668.650262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-20 20:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:12:25.282232725 +0000 UTC m=+2997.368183723" watchObservedRunningTime="2026-07-20 20:12:25.283311011 +0000 UTC m=+2997.369262010"
Jul 20 20:12:26.133524 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:26.133453    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ww9zr_57db0719-7a2e-49bf-9356-8b1ea7b94069/global-pull-secret-syncer/0.log"
Jul 20 20:12:26.258068 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:26.258024    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mb4tg_1e7a94d4-ebe8-4991-92b5-157d5153b6ef/konnectivity-agent/0.log"
Jul 20 20:12:26.302642 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:26.302602    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-227.ec2.internal_e2e50227d65fda4050a8a8dd57637707/haproxy/0.log"
Jul 20 20:12:28.575777 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:28.575730    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 20:12:28.592463 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:28.592438    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-42skd_e4531c13-81bf-480d-b18e-0a31aceb78d6/ovn-acl-logging/0.log"
Jul 20 20:12:31.511281 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:31.510646    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85756d6b4-c2zbv_d82fd48a-0bd3-4367-a994-6079e3cf5897/authorino/0.log"
Jul 20 20:12:31.603218 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:31.603179    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9nzjq_53d4d1e3-1196-4995-a421-44c48295b555/kuadrant-console-plugin/0.log"
Jul 20 20:12:34.096556 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:34.096477    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4hlp8_6094ea31-c721-46b1-9209-e919faff4a00/node-exporter/0.log"
Jul 20 20:12:34.119315 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:34.119280    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4hlp8_6094ea31-c721-46b1-9209-e919faff4a00/kube-rbac-proxy/0.log"
Jul 20 20:12:34.145488 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:34.145399    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4hlp8_6094ea31-c721-46b1-9209-e919faff4a00/init-textfile/0.log"
Jul 20 20:12:34.695546 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:34.695511    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-zjjzb_26599cb7-6a15-4ded-a7ee-14b884d2d9ee/prometheus-operator-admission-webhook/0.log"
Jul 20 20:12:36.697518 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.697478    2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"]
Jul 20 20:12:36.711805 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.711765    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.720802 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.720765    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"]
Jul 20 20:12:36.775387 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.775229    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-sys\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.775387 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.775293    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-lib-modules\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.775387 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.775344    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxt7\" (UniqueName: \"kubernetes.io/projected/d38c8f34-cc86-477c-a60b-f22314e467da-kube-api-access-rsxt7\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.775961 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.775463    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-podres\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.775961 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.775526    2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-proc\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.877016 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.876974    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-podres\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.877825 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.877340    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-podres\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.877962 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.877792    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-proc\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.877962 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.877933    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-sys\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.878088 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.877984    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-lib-modules\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.878088 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.878069    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-sys\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.878300 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.878275    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-proc\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.878412 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.878288    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d38c8f34-cc86-477c-a60b-f22314e467da-lib-modules\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.878697 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.878307    2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxt7\" (UniqueName: \"kubernetes.io/projected/d38c8f34-cc86-477c-a60b-f22314e467da-kube-api-access-rsxt7\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:36.888187 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:36.888162    2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxt7\" (UniqueName: \"kubernetes.io/projected/d38c8f34-cc86-477c-a60b-f22314e467da-kube-api-access-rsxt7\") pod \"perf-node-gather-daemonset-4djnp\" (UID: \"d38c8f34-cc86-477c-a60b-f22314e467da\") " pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:37.029432 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:37.029326    2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:37.209600 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:37.207892    2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"]
Jul 20 20:12:37.211941 ip-10-0-138-227 kubenswrapper[2550]: W0720 20:12:37.211903    2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd38c8f34_cc86_477c_a60b_f22314e467da.slice/crio-210a63442c2621da6e658e2615fad17da30b76fa3dc42da2be4470cc6225d26d WatchSource:0}: Error finding container 210a63442c2621da6e658e2615fad17da30b76fa3dc42da2be4470cc6225d26d: Status 404 returned error can't find the container with id 210a63442c2621da6e658e2615fad17da30b76fa3dc42da2be4470cc6225d26d
Jul 20 20:12:37.342313 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:37.342249    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp" event={"ID":"d38c8f34-cc86-477c-a60b-f22314e467da","Type":"ContainerStarted","Data":"6f1c3c77a396f766047f2f2cb39dcae211c326395491ce5a689d627c21639f62"}
Jul 20 20:12:37.342653 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:37.342342    2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp" event={"ID":"d38c8f34-cc86-477c-a60b-f22314e467da","Type":"ContainerStarted","Data":"210a63442c2621da6e658e2615fad17da30b76fa3dc42da2be4470cc6225d26d"}
Jul 20 20:12:37.342653 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:37.342485    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:37.355987 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:37.355960    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-774c66cdbb-fvh8v_aaafa640-41e2-49f3-a712-840711843ac9/console/0.log"
Jul 20 20:12:37.359227 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:37.359180    2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp" podStartSLOduration=1.3591651310000001 podStartE2EDuration="1.359165131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-20 20:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-20 20:12:37.357851517 +0000 UTC m=+3009.443802515" watchObservedRunningTime="2026-07-20 20:12:37.359165131 +0000 UTC m=+3009.445116128"
Jul 20 20:12:39.117884 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:39.117853    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vlxp5_2fe76da7-11be-40ce-81e9-8e57306a184d/dns/0.log"
Jul 20 20:12:39.158788 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:39.158765    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vlxp5_2fe76da7-11be-40ce-81e9-8e57306a184d/kube-rbac-proxy/0.log"
Jul 20 20:12:39.201506 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:39.201466    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2kzc8_ed7bad58-74fb-44fc-9d4d-f2f04d4a1602/dns-node-resolver/0.log"
Jul 20 20:12:39.851838 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:39.851812    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-54b4h_d8b8de86-0507-4fb5-a218-ef90a0a8100c/node-ca/0.log"
Jul 20 20:12:41.159051 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:41.159016    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7578c486d7-vrkgc_45994ef8-30f2-48c1-8707-44b700e57a2b/kube-auth-proxy/0.log"
Jul 20 20:12:41.345553 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:41.345514    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-4chbp_e19a7680-c1b3-49e7-8e02-e3d48ecba89d/payload-processing/0.log"
Jul 20 20:12:42.086025 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:42.085989    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gk5f7_471d7512-1535-4263-95db-a1fb1e7bd134/serve-healthcheck-canary/0.log"
Jul 20 20:12:42.627092 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:42.627065    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7lfs6_deddaa5c-cfd1-42bf-99e5-4a1f3b434a11/kube-rbac-proxy/0.log"
Jul 20 20:12:42.655476 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:42.655439    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7lfs6_deddaa5c-cfd1-42bf-99e5-4a1f3b434a11/exporter/0.log"
Jul 20 20:12:42.690484 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:42.690454    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7lfs6_deddaa5c-cfd1-42bf-99e5-4a1f3b434a11/extractor/0.log"
Jul 20 20:12:43.361374 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:43.361339    2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-87kz5/perf-node-gather-daemonset-4djnp"
Jul 20 20:12:45.507089 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:45.507056    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-75c5bf7b68-9db78_6de81c58-514d-4d01-9d9b-77e54012c519/manager/0.log"
Jul 20 20:12:45.630743 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:45.630699    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n7b9c_13b6218d-ee95-4de8-8171-26749fc7da1d/manager/0.log"
Jul 20 20:12:45.661049 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:45.661003    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr_76d56700-1f80-437b-9565-9a3d46e7f3d9/manager/0.log"
Jul 20 20:12:45.697529 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:45.697502    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d6d57ff7c-9mjwr_76d56700-1f80-437b-9565-9a3d46e7f3d9/copy-manifests/0.log"
Jul 20 20:12:47.268430 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:47.268347    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c66c54875-9j685_ad9d7768-e440-4657-8d4e-b575372f7323/manager/0.log"
Jul 20 20:12:47.321845 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:47.321813    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7578c486d7-vrkgc_45994ef8-30f2-48c1-8707-44b700e57a2b/kube-auth-proxy/0.log"
Jul 20 20:12:47.427000 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:47.426971    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-4chbp_e19a7680-c1b3-49e7-8e02-e3d48ecba89d/payload-processing/0.log"
Jul 20 20:12:48.840125 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:48.840090    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7578c486d7-vrkgc_45994ef8-30f2-48c1-8707-44b700e57a2b/kube-auth-proxy/0.log"
Jul 20 20:12:48.929708 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:48.929667    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-4chbp_e19a7680-c1b3-49e7-8e02-e3d48ecba89d/payload-processing/0.log"
Jul 20 20:12:53.795638 ip-10-0-138-227 kubenswrapper[2550]: I0720 20:12:53.795595    2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-55bk8_0b013675-0f9f-4383-bb33-cf662c795463/kube-multus/0.log"