Jul 25 01:11:55.783602 ip-10-0-135-99 systemd[1]: Starting Kubernetes Kubelet...
Jul 25 01:11:56.163445 ip-10-0-135-99 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 25 01:11:56.163445 ip-10-0-135-99 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 25 01:11:56.163445 ip-10-0-135-99 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 25 01:11:56.163445 ip-10-0-135-99 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 25 01:11:56.163445 ip-10-0-135-99 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 25 01:11:56.164387 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.164285    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 25 01:11:56.169773 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169757    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 25 01:11:56.169814 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169775    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 25 01:11:56.169814 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169780    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 25 01:11:56.169814 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169785    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 25 01:11:56.169814 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169789    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 25 01:11:56.169814 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169793    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 25 01:11:56.169814 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169798    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 25 01:11:56.169814 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169801    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 25 01:11:56.169814 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169806    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 25 01:11:56.169814 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169810    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 25 01:11:56.169814 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169814    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 25 01:11:56.169814 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169817    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169824    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169829    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169834    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169838    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169843    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169847    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169857    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169861    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169865    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169868    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169873    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169877    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169881    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169884    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169888    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169892    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169895    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169901    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 25 01:11:56.170076 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169906    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169910    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169914    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169919    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169923    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169926    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169930    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169934    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169938    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169941    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169945    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169949    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169953    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169957    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169960    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169965    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169969    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169973    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169977    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169980    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 25 01:11:56.170540 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169984    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169987    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169991    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169995    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.169998    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170003    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170006    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170009    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170013    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170017    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170021    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170025    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170029    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170032    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170036    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170040    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170044    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170048    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170051    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170055    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 25 01:11:56.171070 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170059    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170062    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170066    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170070    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170073    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170077    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170081    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170086    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170090    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170094    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170098    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170102    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170105    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170109    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170113    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170117    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170121    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170124    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170128    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 25 01:11:56.171563 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170132    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170135    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170142    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170146    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170149    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170153    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170157    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170160    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170165    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.170169    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171129    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171139    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171143    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171147    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171151    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171156    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171160    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171164    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171168    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171172    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 25 01:11:56.172020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171177    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171181    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171185    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171189    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171193    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171196    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171200    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171204    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171208    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171212    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171216    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171219    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171224    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171229    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171233    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171236    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171240    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171244    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171247    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 25 01:11:56.172549 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171251    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171257    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171263    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171267    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171271    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171276    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171280    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171284    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171288    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171292    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171297    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171302    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171307    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171311    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171315    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171319    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171336    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171340    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171344    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171348    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 25 01:11:56.173029 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171351    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171355    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171359    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171363    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171366    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171370    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171374    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171378    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171382    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171385    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171389    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171393    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171396    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171400    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171404    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171408    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171411    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171415    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171418    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171422    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 25 01:11:56.173626 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171425    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171430    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171436    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171441    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171445    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171449    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171454    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171460    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171463    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171467    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171471    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171474    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171478    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171482    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171485    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171489    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171493    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171497    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171501    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 25 01:11:56.174130 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171505    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171510    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171515    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171519    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171523    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171527    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171530    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171534    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171537    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171541    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.171545    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172069    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172075    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172079    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172084    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172088    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172092    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172098    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172104    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 25 01:11:56.174630 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172108    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172112    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172116    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172120    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172123    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172127    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172130    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172134    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172138    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172142    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172145    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172149    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172153    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172156    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172160    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172164    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172167    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172171    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172174    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172178    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 25 01:11:56.175182 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172181    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172184    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172188    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172193    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172198    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172202    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172205    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172209    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172214    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172217    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172221    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172225    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172228    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172232    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172236    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172239    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172243    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172247    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172250    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 25 01:11:56.175725 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172254    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172257    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172261    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172265    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172271    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172274    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172278    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172282    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172285    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172289    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172293    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172297    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172301    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172305    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172309    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172312    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172316    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172333    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172337    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172341    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 25 01:11:56.176229 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172345    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172349    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172361    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172366    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172370    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172374    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172378    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172381    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172385    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172390    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172394    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172398    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172401    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172405    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172409    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172413    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172419    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172422    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172426    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172430    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 25 01:11:56.176731 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172433    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172437    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172440    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172444    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172448    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172452    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172455    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172459    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172463    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172467    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172470    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.172474    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172561    2572 flags.go:64] FLAG: --address="0.0.0.0"
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172578    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172586    2572 flags.go:64] FLAG: --anonymous-auth="true"
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172593    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172598    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172603    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172609    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172615    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172620    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 25 01:11:56.177219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172624    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172629    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172634    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172638    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172643    2572 flags.go:64] FLAG: --cgroup-root=""
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172647    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172651    2572 flags.go:64] FLAG: --client-ca-file=""
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172657    2572 flags.go:64] FLAG: --cloud-provider="external"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172661    2572 flags.go:64] FLAG: --cluster-dns="[]"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172667    2572 flags.go:64] FLAG: --cluster-domain=""
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172671    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172675    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172680    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172684    2572 flags.go:64] FLAG: --container-log-max-files="5"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172690    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172694    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172698    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172703    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172707    2572 flags.go:64] FLAG: --contention-profiling="false"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172711    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172715    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172719    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172723    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172729    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 25 01:11:56.177796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172734    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172737    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172741    2572 flags.go:64] FLAG: --enable-load-reader="false"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172754    2572 flags.go:64] FLAG: --enable-server="true"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172759    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172764    2572 flags.go:64] FLAG: --event-burst="100"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172769    2572 flags.go:64] FLAG: --event-qps="50"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172772    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172777    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172782    2572 flags.go:64] FLAG: --eviction-hard=""
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172786    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172790    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172794    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172799    2572 flags.go:64] FLAG: --eviction-soft=""
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172803    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172810    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172814    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172818    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172822    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172826    2572 flags.go:64] FLAG: --fail-swap-on="true"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172830    2572 flags.go:64] FLAG: --feature-gates=""
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172836    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172840    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172845    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172850    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172854    2572 flags.go:64] FLAG: --healthz-port="10248"
Jul 25 01:11:56.178388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172858    2572 flags.go:64] FLAG: --help="false"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172862    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172866    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172870    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172874    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172880    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172884    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172888    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172893    2572 flags.go:64] FLAG: --image-service-endpoint=""
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172898    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172912    2572 flags.go:64] FLAG: --kube-api-burst="100"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172917    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172922    2572 flags.go:64] FLAG: --kube-api-qps="50"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172926    2572 flags.go:64] FLAG: --kube-reserved=""
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172930    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172934    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172938    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172942    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172946    2572 flags.go:64] FLAG: --lock-file=""
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172950    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172954    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172959    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172967    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 25 01:11:56.179039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172971    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172976    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172979    2572 flags.go:64] FLAG: --logging-format="text"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172983    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172988    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172992    2572 flags.go:64] FLAG: --manifest-url=""
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.172996    2572 flags.go:64] FLAG: --manifest-url-header=""
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173001    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173006    2572 flags.go:64] FLAG: --max-open-files="1000000"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173011    2572 flags.go:64] FLAG: --max-pods="110"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173015    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173020    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173024    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173028    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173032    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173037    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173041    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173051    2572 flags.go:64] FLAG: --node-status-max-images="50"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173057    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173061    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173072    2572 flags.go:64] FLAG: --pod-cidr=""
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173077    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173085    2572 flags.go:64] FLAG: --pod-manifest-path=""
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173089    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 25 01:11:56.179607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173093    2572 flags.go:64] FLAG: --pods-per-core="0"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173098    2572 flags.go:64] FLAG: --port="10250"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173102    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173106    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04d247a44fcc6272f"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173111    2572 flags.go:64] FLAG: --qos-reserved=""
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173115    2572 flags.go:64] FLAG: --read-only-port="10255"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173121    2572 flags.go:64] FLAG: --register-node="true"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173125    2572 flags.go:64] FLAG: --register-with-taints=""
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173130    2572 flags.go:64] FLAG: --registry-burst="10"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173134    2572 flags.go:64] FLAG: --registry-qps="5"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173138    2572 flags.go:64] FLAG: --reserved-cpus=""
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173142    2572 flags.go:64] FLAG: --reserved-memory=""
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173147    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173151    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173155    2572 flags.go:64] FLAG: --rotate-certificates="false"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173159    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173163    2572 flags.go:64] FLAG: --runonce="false"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173167    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173171    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173175    2572 flags.go:64] FLAG: --seccomp-default="false"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173179    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173183    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173188    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173192    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173196    2572 flags.go:64] FLAG: --storage-driver-password="root"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173199    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 25 01:11:56.180406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173204    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173210    2572 flags.go:64] FLAG: --storage-driver-user="root"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173214    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173225    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173229    2572 flags.go:64] FLAG: --system-cgroups=""
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173233    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173241    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173244    2572 flags.go:64] FLAG: --tls-cert-file=""
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173255    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173264    2572 flags.go:64] FLAG: --tls-min-version=""
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173269    2572 flags.go:64] FLAG: --tls-private-key-file=""
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173272    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173278    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173283    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173288    2572 flags.go:64] FLAG: --v="2"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173293    2572 flags.go:64] FLAG: --version="false"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173298    2572 flags.go:64] FLAG: --vmodule=""
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173303    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.173308    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173522    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173528    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173532    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173537    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173541    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 25 01:11:56.181337 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173544    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173548    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173552    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173556    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173560    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173563    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173567    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173570    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173574    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173579    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173583    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173587    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173598    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173601    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173605    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173609    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173613    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173616    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173620    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173624    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 25 01:11:56.182139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173629    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173633    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173637    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173640    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173644    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173647    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173651    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173655    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173659    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173663    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173667    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173670    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173674    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173677    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173681    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173685    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173689    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173692    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173695    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173699    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 25 01:11:56.182665 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173703    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173707    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173711    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173715    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173719    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173729    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173733    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173737    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173741    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173744    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173748    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173754    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173760    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173763    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173767    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173770    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173774    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173778    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173782    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173786    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 25 01:11:56.183163 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173790    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173794    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173797    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173800    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173804    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173808    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173813    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173818    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173823    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173827    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173831    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173835    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173839    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173844    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173848    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173851    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173855    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173859    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173870    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 25 01:11:56.183668 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173874    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173878    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173882    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173885    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173889    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173894    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173898    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173901    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173906    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173909    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173913    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173916    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173919    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173923    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.173927    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 25 01:11:56.184141 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.174701    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.182886    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.182901    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.183764    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.183771    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183830    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183835    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183838    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183841    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183844    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183847    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183850    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183853    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183855    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183858    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183861    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183863    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183866    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183868    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183871    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 25 01:11:56.184575 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183874    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183876    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183879    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183882    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183885    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183887    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183890    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183893    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183895    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183898    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183900    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183902    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183905    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183912    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183915    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183917    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183920    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183922    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183924    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183927    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 25 01:11:56.185061 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183930    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183933    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183937    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183940    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183943    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183945    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183948    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183951    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183954    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183956    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183959    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183962    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183965    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183967    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183970    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183973    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183975    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183978    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183981    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 25 01:11:56.185568 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183983    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183986    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183988    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183991    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.183994    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184004    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184007    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184011    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184015    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184018    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184021    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184024    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184026    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184029    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184033    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184036    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184039    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184041    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184044    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 25 01:11:56.186050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184046    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184049    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184051    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184054    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184056    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184059    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184063    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184066    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184068    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184071    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184074    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184076    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184079    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184082    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184084    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184087    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184089    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184092    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184094    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184097    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 25 01:11:56.186537 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184100    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184102    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184105    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184108    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184110    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184113    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.184118    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184236    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184241    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184244    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184247    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184250    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184253    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184255    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 25 01:11:56.187024 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184258    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184260    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184263    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184266    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184268    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184277    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184282    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184285    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184288    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184291    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184293    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184297    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184300    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184302    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184305    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184308    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184310    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184313    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184316    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184319    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 25 01:11:56.187381 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184335    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184338    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184341    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184344    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184346    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184349    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184352    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184354    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184357    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184360    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184362    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184366    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184369    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184371    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184374    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184377    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184380    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184383    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184387    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 25 01:11:56.187922 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184390    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184393    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184395    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184398    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184400    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184403    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184405    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184408    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184410    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184413    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184415    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184418    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184420    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184423    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184425    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184428    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184431    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184433    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184436    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184438    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 25 01:11:56.188515 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184441    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184443    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184446    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184448    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184451    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184453    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184456    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184458    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184463    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184466    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184470    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184472    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184477    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184480    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184483    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184486    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184489    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184492    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184494    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 25 01:11:56.189090 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184497    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184500    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184502    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184505    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184507    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184510    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184512    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184515    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184517    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184520    2572 feature_gate.go:328] unrecognized feature gate: Example
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184522    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184525    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184527    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:56.184530    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.184535    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 25 01:11:56.189577 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.184628    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 25 01:11:56.189940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.187439    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 25 01:11:56.189940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.188892    2572 server.go:1019] "Starting client certificate rotation"
Jul 25 01:11:56.189940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.189011    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 25 01:11:56.189940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.189047    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 25 01:11:56.191959 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.191946    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 25 01:11:56.193135 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.193111    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 25 01:11:56.204073 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.204055    2572 log.go:25] "Validated CRI v1 runtime API"
Jul 25 01:11:56.208351 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.208318    2572 log.go:25] "Validated CRI v1 image API"
Jul 25 01:11:56.209189 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.209174    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 25 01:11:56.213378 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.213311    2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e33b961f-9917-436e-b5d5-a79e9dc1ef68:/dev/nvme0n1p3 ffaf4e3e-0df2-4b84-95f7-0be67d1f7ac7:/dev/nvme0n1p4]
Jul 25 01:11:56.213462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.213372    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 25 01:11:56.219119 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.219101    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 25 01:11:56.219799 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.219701    2572 manager.go:216] Machine: {Timestamp:2026-07-25 01:11:56.218420112 +0000 UTC m=+0.328229387 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083949 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec283bb080203b86516efba19f97cace SystemUUID:ec283bb0-8020-3b86-516e-fba19f97cace BootID:fd0158b6-cac4-4c3a-92e8-b5c9b38ba12d Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:de:98:39:dc:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:de:98:39:dc:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:4b:b0:84:5c:79 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 25 01:11:56.219799 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.219794    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 25 01:11:56.219903 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.219891    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 25 01:11:56.223310 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.223287    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 25 01:11:56.223449 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.223313    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-99.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 25 01:11:56.224491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.224480    2572 topology_manager.go:138] "Creating topology manager with none policy"
Jul 25 01:11:56.224531 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.224493    2572 container_manager_linux.go:309] "Creating device plugin manager"
Jul 25 01:11:56.224531 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.224501    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 25 01:11:56.224531 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.224514    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 25 01:11:56.225307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.225298    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 25 01:11:56.227216 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.227206    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 25 01:11:56.227313 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.227305    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 25 01:11:56.227352 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.227346    2572 kubelet.go:486] "Attempting to sync node with API server"
Jul 25 01:11:56.227384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.227356    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 25 01:11:56.227384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.227376    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 25 01:11:56.227384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.227384    2572 kubelet.go:398] "Adding apiserver pod source"
Jul 25 01:11:56.227474 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.227393    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 25 01:11:56.227677 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.227666    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 25 01:11:56.227711 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.227684    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 25 01:11:56.229816 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.229795    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 25 01:11:56.231651 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.231637    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 25 01:11:56.231717 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.231666    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 25 01:11:56.233342 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233315    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 25 01:11:56.233420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233347    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 25 01:11:56.233420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233357    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 25 01:11:56.233420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233363    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 25 01:11:56.233420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233372    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 25 01:11:56.233420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233379    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 25 01:11:56.233420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233387    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 25 01:11:56.233420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233393    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 25 01:11:56.233420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233399    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 25 01:11:56.233420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233405    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 25 01:11:56.233420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233413    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 25 01:11:56.233420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.233419    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 25 01:11:56.234166 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.234151    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 25 01:11:56.234205 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.234169    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 25 01:11:56.236107 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.236082    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 25 01:11:56.236176 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.236115    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 25 01:11:56.236739 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.236720    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-99.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 25 01:11:56.237123 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.237111    2572 server.go:1268] "Started kubelet"
Jul 25 01:11:56.237197 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.237175    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 25 01:11:56.237298 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.237243    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 25 01:11:56.237391 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.237360    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 25 01:11:56.237808 ip-10-0-135-99 systemd[1]: Started Kubernetes Kubelet.
Jul 25 01:11:56.239214 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.239193    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 25 01:11:56.240227 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.239874    2572 server.go:310] "Adding debug handlers to kubelet server"
Jul 25 01:11:56.243957 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.243935    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 25 01:11:56.248154 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.248136    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 25 01:11:56.248697 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.248681    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 25 01:11:56.248821 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.248785    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-99.ec2.internal\" not found"
Jul 25 01:11:56.249374 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.249359    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 25 01:11:56.249481 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.249462    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 25 01:11:56.249591 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.249359    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 25 01:11:56.249591 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.249566    2572 reconstruct.go:97] "Volume reconstruction finished"
Jul 25 01:11:56.249591 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.249575    2572 reconciler.go:29] "Reconciler: start to sync state"
Jul 25 01:11:56.249736 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.249690    2572 factory.go:55] Registering systemd factory
Jul 25 01:11:56.249736 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.249704    2572 factory.go:223] Registration of the systemd container factory successfully
Jul 25 01:11:56.250015 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.250000    2572 factory.go:153] Registering CRI-O factory
Jul 25 01:11:56.250015 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.250014    2572 factory.go:223] Registration of the crio container factory successfully
Jul 25 01:11:56.250128 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.250056    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 25 01:11:56.250128 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.250077    2572 factory.go:103] Registering Raw factory
Jul 25 01:11:56.250128 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.250088    2572 manager.go:1196] Started watching for new ooms in manager
Jul 25 01:11:56.250500 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.250481    2572 manager.go:349] Starting recovery of all containers
Jul 25 01:11:56.254595 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.254563    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 25 01:11:56.254952 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.254927    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-99.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 25 01:11:56.255643 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.254634    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-99.ec2.internal.18c563373e043d97  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-99.ec2.internal,UID:ip-10-0-135-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-99.ec2.internal,},FirstTimestamp:2026-07-25 01:11:56.237086103 +0000 UTC m=+0.346895377,LastTimestamp:2026-07-25 01:11:56.237086103 +0000 UTC m=+0.346895377,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-99.ec2.internal,}"
Jul 25 01:11:56.259660 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.259644    2572 manager.go:354] Recovery completed
Jul 25 01:11:56.264052 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.264040    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 25 01:11:56.266381 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.266361    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 25 01:11:56.266478 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.266394    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 25 01:11:56.266478 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.266405    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientPID"
Jul 25 01:11:56.266949 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.266932    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 25 01:11:56.266949 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.266946    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 25 01:11:56.267073 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.266961    2572 state_mem.go:36] "Initialized new in-memory state store"
Jul 25 01:11:56.267073 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.266974    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 25 01:11:56.267663 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.267636    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 25 01:11:56.267663 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.267657    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 25 01:11:56.267663 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.267666    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientPID"
Jul 25 01:11:56.268424 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.268364    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-99.ec2.internal.18c563373fc33e12  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-99.ec2.internal,UID:ip-10-0-135-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-99.ec2.internal,},FirstTimestamp:2026-07-25 01:11:56.266380818 +0000 UTC m=+0.376190094,LastTimestamp:2026-07-25 01:11:56.266380818 +0000 UTC m=+0.376190094,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-99.ec2.internal,}"
Jul 25 01:11:56.268601 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.268588    2572 policy_none.go:49] "None policy: Start"
Jul 25 01:11:56.268635 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.268608    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 25 01:11:56.268635 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.268622    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 25 01:11:56.269556 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.269545    2572 policy_none.go:47] "Start"
Jul 25 01:11:56.276510 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.276446    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-99.ec2.internal.18c563373fc38936  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-99.ec2.internal,UID:ip-10-0-135-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-99.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-99.ec2.internal,},FirstTimestamp:2026-07-25 01:11:56.266400054 +0000 UTC m=+0.376209329,LastTimestamp:2026-07-25 01:11:56.266400054 +0000 UTC m=+0.376209329,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-99.ec2.internal,}"
Jul 25 01:11:56.285072 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.285019    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-99.ec2.internal.18c563373fc3ac1d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-99.ec2.internal,UID:ip-10-0-135-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-99.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-99.ec2.internal,},FirstTimestamp:2026-07-25 01:11:56.266408989 +0000 UTC m=+0.376218263,LastTimestamp:2026-07-25 01:11:56.266408989 +0000 UTC m=+0.376218263,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-99.ec2.internal,}"
Jul 25 01:11:56.290398 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.290367    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w2f6t"
Jul 25 01:11:56.295569 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.295422    2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-99.ec2.internal.18c563373fc33e12\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-99.ec2.internal.18c563373fc33e12  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-99.ec2.internal,UID:ip-10-0-135-99.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-99.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-99.ec2.internal,},FirstTimestamp:2026-07-25 01:11:56.266380818 +0000 UTC m=+0.376190094,LastTimestamp:2026-07-25 01:11:56.267649533 +0000 UTC m=+0.377458809,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-99.ec2.internal,}"
Jul 25 01:11:56.298990 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.298975    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w2f6t"
Jul 25 01:11:56.302929 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.302916    2572 manager.go:337] "Starting Device Plugin manager"
Jul 25 01:11:56.302995 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.302946    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 25 01:11:56.302995 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.302958    2572 server.go:87] "Starting device plugin registration server"
Jul 25 01:11:56.303165 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.303154    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 25 01:11:56.303202 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.303166    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 25 01:11:56.303286 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.303271    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 25 01:11:56.303413 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.303398    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 25 01:11:56.303413 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.303410    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 25 01:11:56.303796 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.303774    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 25 01:11:56.303898 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.303819    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-99.ec2.internal\" not found"
Jul 25 01:11:56.345501 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.345479    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 25 01:11:56.346685 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.346667    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 25 01:11:56.346746 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.346707    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 25 01:11:56.346746 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.346729    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 25 01:11:56.346838 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.346761    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 25 01:11:56.349309 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.349289    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 25 01:11:56.403409 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.403386    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 25 01:11:56.404201 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.404185    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 25 01:11:56.404270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.404215    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 25 01:11:56.404270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.404226    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientPID"
Jul 25 01:11:56.404270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.404251    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.414380 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.414337    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.414380 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.414359    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-99.ec2.internal\": node \"ip-10-0-135-99.ec2.internal\" not found"
Jul 25 01:11:56.429241 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.429223    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-99.ec2.internal\" not found"
Jul 25 01:11:56.447854 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.447833    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-99.ec2.internal"]
Jul 25 01:11:56.447939 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.447895    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 25 01:11:56.449625 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.449606    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 25 01:11:56.449710 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.449637    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 25 01:11:56.449710 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.449647    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientPID"
Jul 25 01:11:56.450773 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.450758    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dc2e8379747207d092a55285ea986ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal\" (UID: \"0dc2e8379747207d092a55285ea986ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.450815 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.450782    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dc2e8379747207d092a55285ea986ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal\" (UID: \"0dc2e8379747207d092a55285ea986ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.450961 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.450949    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 25 01:11:56.451088 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.451077    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.451122 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.451103    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 25 01:11:56.451671 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.451656    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 25 01:11:56.451728 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.451687    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 25 01:11:56.451728 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.451698    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientPID"
Jul 25 01:11:56.451787 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.451656    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 25 01:11:56.451787 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.451764    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 25 01:11:56.451787 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.451776    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientPID"
Jul 25 01:11:56.452830 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.452817    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.452870 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.452840    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 25 01:11:56.453994 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.453979    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientMemory"
Jul 25 01:11:56.454087 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.454007    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasNoDiskPressure"
Jul 25 01:11:56.454087 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.454021    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeHasSufficientPID"
Jul 25 01:11:56.465867 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.465850    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-99.ec2.internal\" not found" node="ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.469704 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.469688    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-99.ec2.internal\" not found" node="ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.530246 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.530215    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-99.ec2.internal\" not found"
Jul 25 01:11:56.551898 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.551870    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dc2e8379747207d092a55285ea986ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal\" (UID: \"0dc2e8379747207d092a55285ea986ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.552002 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.551903    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dc2e8379747207d092a55285ea986ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal\" (UID: \"0dc2e8379747207d092a55285ea986ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.552002 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.551946    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47c77adca673d868115fa74ace1d7b3f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-99.ec2.internal\" (UID: \"47c77adca673d868115fa74ace1d7b3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.552002 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.551981    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dc2e8379747207d092a55285ea986ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal\" (UID: \"0dc2e8379747207d092a55285ea986ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.552101 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.551982    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dc2e8379747207d092a55285ea986ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal\" (UID: \"0dc2e8379747207d092a55285ea986ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.630474 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.630455    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-99.ec2.internal\" not found"
Jul 25 01:11:56.652771 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.652752    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47c77adca673d868115fa74ace1d7b3f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-99.ec2.internal\" (UID: \"47c77adca673d868115fa74ace1d7b3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.652821 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.652811    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/47c77adca673d868115fa74ace1d7b3f-config\") pod \"kube-apiserver-proxy-ip-10-0-135-99.ec2.internal\" (UID: \"47c77adca673d868115fa74ace1d7b3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.731383 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.731358    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-99.ec2.internal\" not found"
Jul 25 01:11:56.768555 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.768534    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.773666 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:56.773474    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:56.832546 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.832517    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-99.ec2.internal\" not found"
Jul 25 01:11:56.932832 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:56.932806    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-99.ec2.internal\" not found"
Jul 25 01:11:57.035086 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.035000    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-99.ec2.internal\" not found"
Jul 25 01:11:57.056868 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.056840    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 25 01:11:57.135544 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.135510    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-99.ec2.internal\" not found"
Jul 25 01:11:57.188543 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.188522    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 25 01:11:57.188948 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.188697    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 25 01:11:57.188948 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.188712    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 25 01:11:57.200833 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.200811    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 25 01:11:57.228174 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.228147    2572 apiserver.go:52] "Watching apiserver"
Jul 25 01:11:57.237813 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.237786    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 25 01:11:57.238171 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.238150    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-rk9b7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4","openshift-image-registry/node-ca-jd9wg","openshift-multus/multus-bt57h","openshift-network-operator/iptables-alerter-9x2v2","openshift-ovn-kubernetes/ovnkube-node-vwkth","kube-system/global-pull-secret-syncer-hkv52","openshift-cluster-node-tuning-operator/tuned-mrk86","openshift-dns/node-resolver-fwsm2","openshift-multus/multus-additional-cni-plugins-k6nrz","openshift-multus/network-metrics-daemon-nmt2v","openshift-network-diagnostics/network-check-target-qqdd4"]
Jul 25 01:11:57.239551 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.239532    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:11:57.240679 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.240659    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.241809 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.241788    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9x2v2"
Jul 25 01:11:57.241918 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.241826    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 25 01:11:57.241918 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.241842    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 25 01:11:57.242169 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.242155    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xqwf7\""
Jul 25 01:11:57.242871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.242849    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jwrfm\""
Jul 25 01:11:57.242969 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.242876    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 25 01:11:57.242969 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.242945    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 25 01:11:57.243115 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.243097    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 25 01:11:57.243233 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.243210    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.244162 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.244137    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 25 01:11:57.244162 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.244156    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 25 01:11:57.244326 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.244215    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 25 01:11:57.244326 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.244219    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fbkn2\""
Jul 25 01:11:57.244581 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.244483    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fwsm2"
Jul 25 01:11:57.245330 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.245312    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 25 01:11:57.245478 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.245462    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 25 01:11:57.245561 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.245466    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 25 01:11:57.245612 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.245602    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n9k86\""
Jul 25 01:11:57.245769 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.245746    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 25 01:11:57.245769 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.245760    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 25 01:11:57.245886 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.245757    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 25 01:11:57.245994 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.245964    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.246570 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.246545    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 25 01:11:57.246682 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.246658    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bnqsg\""
Jul 25 01:11:57.246937 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.246922    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 25 01:11:57.247214 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.247200    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:11:57.247291 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.247272    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:11:57.248228 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.248212    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 25 01:11:57.248426 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.248392    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bgpx6\""
Jul 25 01:11:57.248526 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.248462    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 25 01:11:57.248526 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.248498    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 25 01:11:57.248690 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.248523    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 25 01:11:57.248690 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.248393    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jd9wg"
Jul 25 01:11:57.248690 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.248619    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 25 01:11:57.248892 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.248709    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 25 01:11:57.248892 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.248859    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:57.249789 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.249769    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.250632 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.250615    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 25 01:11:57.250741 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.250726    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 25 01:11:57.251054 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.251024    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 25 01:11:57.251174 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.251161    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:57.251232 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.251216    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:11:57.251271 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.251227    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l5jpd\""
Jul 25 01:11:57.251974 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.251956    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 25 01:11:57.252028 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.251966    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pjstm\""
Jul 25 01:11:57.252424 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.252406    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.253953 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.253936    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:11:57.254029 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.253989    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:11:57.254465 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.254447    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 25 01:11:57.254549 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.254536    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 25 01:11:57.254950 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.254935    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ds68h\""
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.256659    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-hostroot\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.256707    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj6nk\" (UniqueName: \"kubernetes.io/projected/036736e7-e135-4c08-be7a-8ac7d1975cdf-kube-api-access-wj6nk\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.256743    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.256775    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d9d7e4de-3c40-43fd-a2c1-67117b28520e-agent-certs\") pod \"konnectivity-agent-rk9b7\" (UID: \"d9d7e4de-3c40-43fd-a2c1-67117b28520e\") " pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.256859    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-etc-openvswitch\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.256913    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-env-overrides\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.256948    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373f9d31-a96b-4a50-b460-0716a102570f-serviceca\") pod \"node-ca-jd9wg\" (UID: \"373f9d31-a96b-4a50-b460-0716a102570f\") " pod="openshift-image-registry/node-ca-jd9wg"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.256973    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-sys-fs\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257038    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-kubelet\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257071    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-var-lib-openvswitch\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257104    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257135    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-registration-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257169    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2x6\" (UniqueName: \"kubernetes.io/projected/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-kube-api-access-pm2x6\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257201    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-node-log\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.257798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-cni-netd\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257270    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373f9d31-a96b-4a50-b460-0716a102570f-host\") pod \"node-ca-jd9wg\" (UID: \"373f9d31-a96b-4a50-b460-0716a102570f\") " pod="openshift-image-registry/node-ca-jd9wg"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257311    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-run-k8s-cni-cncf-io\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257341    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-daemon-config\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257372    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-systemd-units\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257416    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0889cc33-fc59-4e79-a109-6e98bf53cc58-hosts-file\") pod \"node-resolver-fwsm2\" (UID: \"0889cc33-fc59-4e79-a109-6e98bf53cc58\") " pod="openshift-dns/node-resolver-fwsm2"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257448    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0889cc33-fc59-4e79-a109-6e98bf53cc58-tmp-dir\") pod \"node-resolver-fwsm2\" (UID: \"0889cc33-fc59-4e79-a109-6e98bf53cc58\") " pod="openshift-dns/node-resolver-fwsm2"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257476    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-run-netns\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257500    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-run-ovn\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257529    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-ovnkube-config\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.257577    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rrnp\" (UniqueName: \"kubernetes.io/projected/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-kube-api-access-4rrnp\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258183    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/036736e7-e135-4c08-be7a-8ac7d1975cdf-cni-binary-copy\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258236    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-socket-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258295    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/036736e7-e135-4c08-be7a-8ac7d1975cdf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258363    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-etc-selinux\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258428    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c5d45f7-545f-4139-a706-dc19d908743f-iptables-alerter-script\") pod \"iptables-alerter-9x2v2\" (UID: \"2c5d45f7-545f-4139-a706-dc19d908743f\") " pod="openshift-network-operator/iptables-alerter-9x2v2"
Jul 25 01:11:57.258640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258468    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-system-cni-dir\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258498    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-var-lib-kubelet\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258527    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-run-openvswitch\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258571    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258602    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhghh\" (UniqueName: \"kubernetes.io/projected/0889cc33-fc59-4e79-a109-6e98bf53cc58-kube-api-access-fhghh\") pod \"node-resolver-fwsm2\" (UID: \"0889cc33-fc59-4e79-a109-6e98bf53cc58\") " pod="openshift-dns/node-resolver-fwsm2"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258669    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-slash\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgwqt\" (UniqueName: \"kubernetes.io/projected/373f9d31-a96b-4a50-b460-0716a102570f-kube-api-access-wgwqt\") pod \"node-ca-jd9wg\" (UID: \"373f9d31-a96b-4a50-b460-0716a102570f\") " pod="openshift-image-registry/node-ca-jd9wg"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-cni-dir\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258776    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-cnibin\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258800    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/166e0d16-c9f0-4993-b0de-e1cc91fd993a-cni-binary-copy\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258829    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-run-netns\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d9d7e4de-3c40-43fd-a2c1-67117b28520e-konnectivity-ca\") pod \"konnectivity-agent-rk9b7\" (UID: \"d9d7e4de-3c40-43fd-a2c1-67117b28520e\") " pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258904    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-cnibin\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258931    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/036736e7-e135-4c08-be7a-8ac7d1975cdf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258963    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96s5\" (UniqueName: \"kubernetes.io/projected/0062e4c2-a9d5-4631-b2f8-a5e223b85264-kube-api-access-n96s5\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.258991    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-conf-dir\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.259382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259023    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-device-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259054    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c5d45f7-545f-4139-a706-dc19d908743f-host-slash\") pod \"iptables-alerter-9x2v2\" (UID: \"2c5d45f7-545f-4139-a706-dc19d908743f\") " pod="openshift-network-operator/iptables-alerter-9x2v2"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259127    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-log-socket\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259182    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-cni-bin\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259242    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-ovn-node-metrics-cert\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259282    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-os-release\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259320    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvd8w\" (UniqueName: \"kubernetes.io/projected/2c5d45f7-545f-4139-a706-dc19d908743f-kube-api-access-zvd8w\") pod \"iptables-alerter-9x2v2\" (UID: \"2c5d45f7-545f-4139-a706-dc19d908743f\") " pod="openshift-network-operator/iptables-alerter-9x2v2"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259395    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-socket-dir-parent\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259436    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-var-lib-cni-multus\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259471    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-etc-kubernetes\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259512    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-run-systemd\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259546    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259578    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-var-lib-cni-bin\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-run-multus-certs\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259657    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbdnr\" (UniqueName: \"kubernetes.io/projected/166e0d16-c9f0-4993-b0de-e1cc91fd993a-kube-api-access-kbdnr\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259689    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-ovnkube-script-lib\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.260235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259720    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-system-cni-dir\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.261018 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.259752    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-os-release\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.261018 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.260189    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 25 01:11:57.261018 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.260808    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 25 01:11:57.261161 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.261061    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal"]
Jul 25 01:11:57.261161 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.261115    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-99.ec2.internal"
Jul 25 01:11:57.267001 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.266981    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 25 01:11:57.267352 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.267335    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-99.ec2.internal"]
Jul 25 01:11:57.277343 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.277324    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l2bcm"
Jul 25 01:11:57.287002 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.286979    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l2bcm"
Jul 25 01:11:57.288063 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:57.288031    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc2e8379747207d092a55285ea986ce.slice/crio-4d5ba0c57b29340d3a3c1faa87ec0ba6451ea5dee7df6a74ec24d37326d4b1e0 WatchSource:0}: Error finding container 4d5ba0c57b29340d3a3c1faa87ec0ba6451ea5dee7df6a74ec24d37326d4b1e0: Status 404 returned error can't find the container with id 4d5ba0c57b29340d3a3c1faa87ec0ba6451ea5dee7df6a74ec24d37326d4b1e0
Jul 25 01:11:57.288898 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:57.288877    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c77adca673d868115fa74ace1d7b3f.slice/crio-3b3e6b2f0d1a4c528be5689408983e5a1701288489ebe7f4cfca8c733af22702 WatchSource:0}: Error finding container 3b3e6b2f0d1a4c528be5689408983e5a1701288489ebe7f4cfca8c733af22702: Status 404 returned error can't find the container with id 3b3e6b2f0d1a4c528be5689408983e5a1701288489ebe7f4cfca8c733af22702
Jul 25 01:11:57.294331 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.294312    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 25 01:11:57.302786 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.302742    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-24 01:06:56 +0000 UTC" deadline="2028-04-19 13:56:03.61957579 +0000 UTC"
Jul 25 01:11:57.302870 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.302791    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15228h44m6.316794381s"
Jul 25 01:11:57.349461 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.349427    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal" event={"ID":"0dc2e8379747207d092a55285ea986ce","Type":"ContainerStarted","Data":"4d5ba0c57b29340d3a3c1faa87ec0ba6451ea5dee7df6a74ec24d37326d4b1e0"}
Jul 25 01:11:57.350359 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.350330    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 25 01:11:57.350462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.350410    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-99.ec2.internal" event={"ID":"47c77adca673d868115fa74ace1d7b3f","Type":"ContainerStarted","Data":"3b3e6b2f0d1a4c528be5689408983e5a1701288489ebe7f4cfca8c733af22702"}
Jul 25 01:11:57.360513 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360492    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.360563 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360515    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhghh\" (UniqueName: \"kubernetes.io/projected/0889cc33-fc59-4e79-a109-6e98bf53cc58-kube-api-access-fhghh\") pod \"node-resolver-fwsm2\" (UID: \"0889cc33-fc59-4e79-a109-6e98bf53cc58\") " pod="openshift-dns/node-resolver-fwsm2"
Jul 25 01:11:57.360563 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360537    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-sys\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.360563 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360559    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-slash\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.360732 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360576    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgwqt\" (UniqueName: \"kubernetes.io/projected/373f9d31-a96b-4a50-b460-0716a102570f-kube-api-access-wgwqt\") pod \"node-ca-jd9wg\" (UID: \"373f9d31-a96b-4a50-b460-0716a102570f\") " pod="openshift-image-registry/node-ca-jd9wg"
Jul 25 01:11:57.360732 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360616    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-cni-dir\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.360732 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360647    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-slash\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.360732 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360656    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-cnibin\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.360732 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360683    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/166e0d16-c9f0-4993-b0de-e1cc91fd993a-cni-binary-copy\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.360732 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360712    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-run-netns\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360729    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360737    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-sysctl-conf\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360737    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-cnibin\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360793    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea4887a5-60be-4a3f-9580-61202226536e-etc-tuned\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360797    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-run-netns\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360821    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d9d7e4de-3c40-43fd-a2c1-67117b28520e-konnectivity-ca\") pod \"konnectivity-agent-rk9b7\" (UID: \"d9d7e4de-3c40-43fd-a2c1-67117b28520e\") " pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360875    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-cnibin\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360904    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-cni-dir\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360906    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/036736e7-e135-4c08-be7a-8ac7d1975cdf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360923    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-cnibin\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360947    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96s5\" (UniqueName: \"kubernetes.io/projected/0062e4c2-a9d5-4631-b2f8-a5e223b85264-kube-api-access-n96s5\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.360987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.360974    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-conf-dir\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361003    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-host\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361028    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-device-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361052    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c5d45f7-545f-4139-a706-dc19d908743f-host-slash\") pod \"iptables-alerter-9x2v2\" (UID: \"2c5d45f7-545f-4139-a706-dc19d908743f\") " pod="openshift-network-operator/iptables-alerter-9x2v2"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361067    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-conf-dir\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361099    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c5d45f7-545f-4139-a706-dc19d908743f-host-slash\") pod \"iptables-alerter-9x2v2\" (UID: \"2c5d45f7-545f-4139-a706-dc19d908743f\") " pod="openshift-network-operator/iptables-alerter-9x2v2"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361110    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-device-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361162    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-log-socket\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361216    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-log-socket\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361250    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-cni-bin\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361276    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-ovn-node-metrics-cert\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361276    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/166e0d16-c9f0-4993-b0de-e1cc91fd993a-cni-binary-copy\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361303    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-os-release\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361307    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-cni-bin\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361322    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d9d7e4de-3c40-43fd-a2c1-67117b28520e-konnectivity-ca\") pod \"konnectivity-agent-rk9b7\" (UID: \"d9d7e4de-3c40-43fd-a2c1-67117b28520e\") " pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361330    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvd8w\" (UniqueName: \"kubernetes.io/projected/2c5d45f7-545f-4139-a706-dc19d908743f-kube-api-access-zvd8w\") pod \"iptables-alerter-9x2v2\" (UID: \"2c5d45f7-545f-4139-a706-dc19d908743f\") " pod="openshift-network-operator/iptables-alerter-9x2v2"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361379    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-socket-dir-parent\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.361462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361436    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-socket-dir-parent\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361446    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-os-release\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361487    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-var-lib-cni-multus\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361491    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/036736e7-e135-4c08-be7a-8ac7d1975cdf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361515    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-etc-kubernetes\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361541    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-run-systemd\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361552    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-var-lib-cni-multus\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361584    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-etc-kubernetes\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361591    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-run-systemd\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361621    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361623    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361665    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361648    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-var-lib-cni-bin\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361672    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-var-lib-cni-bin\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361707    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-run-multus-certs\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361732    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbdnr\" (UniqueName: \"kubernetes.io/projected/166e0d16-c9f0-4993-b0de-e1cc91fd993a-kube-api-access-kbdnr\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361767    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-run-multus-certs\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrfkm\" (UniqueName: \"kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm\") pod \"network-check-target-qqdd4\" (UID: \"df43e33a-d9bd-4d2e-9a95-f7487dd3c442\") " pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:11:57.362215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361814    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-kubelet-config\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361831    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-dbus\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361849    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-ovnkube-script-lib\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361866    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-system-cni-dir\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361893    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-os-release\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361918    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-hostroot\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361944    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361941    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/036736e7-e135-4c08-be7a-8ac7d1975cdf-system-cni-dir\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361969    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-var-lib-kubelet\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.361979    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-os-release\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362005    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-hostroot\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362010    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj6nk\" (UniqueName: \"kubernetes.io/projected/036736e7-e135-4c08-be7a-8ac7d1975cdf-kube-api-access-wj6nk\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362046    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362072    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-sysctl-d\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362092    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-run\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362115    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzw4d\" (UniqueName: \"kubernetes.io/projected/ea4887a5-60be-4a3f-9580-61202226536e-kube-api-access-rzw4d\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362125    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.363178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362135    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d9d7e4de-3c40-43fd-a2c1-67117b28520e-agent-certs\") pod \"konnectivity-agent-rk9b7\" (UID: \"d9d7e4de-3c40-43fd-a2c1-67117b28520e\") " pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362183    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-kubernetes\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362232    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-lib-modules\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362262    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea4887a5-60be-4a3f-9580-61202226536e-tmp\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362291    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-etc-openvswitch\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362356    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-env-overrides\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362360    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-etc-openvswitch\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362383    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373f9d31-a96b-4a50-b460-0716a102570f-serviceca\") pod \"node-ca-jd9wg\" (UID: \"373f9d31-a96b-4a50-b460-0716a102570f\") " pod="openshift-image-registry/node-ca-jd9wg"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362430    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-sys-fs\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362457    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362471    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-ovnkube-script-lib\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362514    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-systemd\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362549    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-kubelet\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362577    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-var-lib-openvswitch\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.362590    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362616    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362664    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-sys-fs\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.364104 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.362673    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs podName:1f0a939b-732b-4313-b0f9-2ce5feb9cc5c nodeName:}" failed. No retries permitted until 2026-07-25 01:11:57.862633429 +0000 UTC m=+1.972442709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs") pod "network-metrics-daemon-nmt2v" (UID: "1f0a939b-732b-4313-b0f9-2ce5feb9cc5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362704    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-registration-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362749    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-kubelet\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362757    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-env-overrides\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362750    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2x6\" (UniqueName: \"kubernetes.io/projected/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-kube-api-access-pm2x6\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362800    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-node-log\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362813    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-registration-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362845    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-cni-netd\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362850    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362892    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-var-lib-openvswitch\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362912    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/373f9d31-a96b-4a50-b460-0716a102570f-serviceca\") pod \"node-ca-jd9wg\" (UID: \"373f9d31-a96b-4a50-b460-0716a102570f\") " pod="openshift-image-registry/node-ca-jd9wg"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362923    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-node-log\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362953    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373f9d31-a96b-4a50-b460-0716a102570f-host\") pod \"node-ca-jd9wg\" (UID: \"373f9d31-a96b-4a50-b460-0716a102570f\") " pod="openshift-image-registry/node-ca-jd9wg"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.362959    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-host-cni-netd\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363015    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-run-k8s-cni-cncf-io\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363043    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-daemon-config\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363049    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-run-k8s-cni-cncf-io\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.364762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363016    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/373f9d31-a96b-4a50-b460-0716a102570f-host\") pod \"node-ca-jd9wg\" (UID: \"373f9d31-a96b-4a50-b460-0716a102570f\") " pod="openshift-image-registry/node-ca-jd9wg"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363073    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-modprobe-d\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-systemd-units\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363145    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0889cc33-fc59-4e79-a109-6e98bf53cc58-hosts-file\") pod \"node-resolver-fwsm2\" (UID: \"0889cc33-fc59-4e79-a109-6e98bf53cc58\") " pod="openshift-dns/node-resolver-fwsm2"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363187    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0889cc33-fc59-4e79-a109-6e98bf53cc58-tmp-dir\") pod \"node-resolver-fwsm2\" (UID: \"0889cc33-fc59-4e79-a109-6e98bf53cc58\") " pod="openshift-dns/node-resolver-fwsm2"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363214    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-run-netns\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363221    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-systemd-units\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363240    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-run-ovn\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363321    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0889cc33-fc59-4e79-a109-6e98bf53cc58-hosts-file\") pod \"node-resolver-fwsm2\" (UID: \"0889cc33-fc59-4e79-a109-6e98bf53cc58\") " pod="openshift-dns/node-resolver-fwsm2"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363381    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-ovnkube-config\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363453    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rrnp\" (UniqueName: \"kubernetes.io/projected/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-kube-api-access-4rrnp\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363482    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-run-netns\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363531    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/036736e7-e135-4c08-be7a-8ac7d1975cdf-cni-binary-copy\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363561    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-socket-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363550    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0889cc33-fc59-4e79-a109-6e98bf53cc58-tmp-dir\") pod \"node-resolver-fwsm2\" (UID: \"0889cc33-fc59-4e79-a109-6e98bf53cc58\") " pod="openshift-dns/node-resolver-fwsm2"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363577    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/166e0d16-c9f0-4993-b0de-e1cc91fd993a-multus-daemon-config\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363611    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/036736e7-e135-4c08-be7a-8ac7d1975cdf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.365247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363631    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-run-ovn\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363658    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-etc-selinux\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363697    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-socket-dir\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363740    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c5d45f7-545f-4139-a706-dc19d908743f-iptables-alerter-script\") pod \"iptables-alerter-9x2v2\" (UID: \"2c5d45f7-545f-4139-a706-dc19d908743f\") " pod="openshift-network-operator/iptables-alerter-9x2v2"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363759    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0062e4c2-a9d5-4631-b2f8-a5e223b85264-etc-selinux\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363771    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-system-cni-dir\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-var-lib-kubelet\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363820    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-system-cni-dir\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363831    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-sysconfig\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-run-openvswitch\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363924    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/166e0d16-c9f0-4993-b0de-e1cc91fd993a-host-var-lib-kubelet\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.363929    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-run-openvswitch\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.364097    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/036736e7-e135-4c08-be7a-8ac7d1975cdf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.364137    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-ovnkube-config\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.364346    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c5d45f7-545f-4139-a706-dc19d908743f-iptables-alerter-script\") pod \"iptables-alerter-9x2v2\" (UID: \"2c5d45f7-545f-4139-a706-dc19d908743f\") " pod="openshift-network-operator/iptables-alerter-9x2v2"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.364646    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/036736e7-e135-4c08-be7a-8ac7d1975cdf-cni-binary-copy\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.365904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.365851    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-ovn-node-metrics-cert\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.366749 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.365929    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d9d7e4de-3c40-43fd-a2c1-67117b28520e-agent-certs\") pod \"konnectivity-agent-rk9b7\" (UID: \"d9d7e4de-3c40-43fd-a2c1-67117b28520e\") " pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:11:57.368695 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.368672    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgwqt\" (UniqueName: \"kubernetes.io/projected/373f9d31-a96b-4a50-b460-0716a102570f-kube-api-access-wgwqt\") pod \"node-ca-jd9wg\" (UID: \"373f9d31-a96b-4a50-b460-0716a102570f\") " pod="openshift-image-registry/node-ca-jd9wg"
Jul 25 01:11:57.369358 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.369332    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvd8w\" (UniqueName: \"kubernetes.io/projected/2c5d45f7-545f-4139-a706-dc19d908743f-kube-api-access-zvd8w\") pod \"iptables-alerter-9x2v2\" (UID: \"2c5d45f7-545f-4139-a706-dc19d908743f\") " pod="openshift-network-operator/iptables-alerter-9x2v2"
Jul 25 01:11:57.369514 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.369498    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhghh\" (UniqueName: \"kubernetes.io/projected/0889cc33-fc59-4e79-a109-6e98bf53cc58-kube-api-access-fhghh\") pod \"node-resolver-fwsm2\" (UID: \"0889cc33-fc59-4e79-a109-6e98bf53cc58\") " pod="openshift-dns/node-resolver-fwsm2"
Jul 25 01:11:57.373181 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.373159    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbdnr\" (UniqueName: \"kubernetes.io/projected/166e0d16-c9f0-4993-b0de-e1cc91fd993a-kube-api-access-kbdnr\") pod \"multus-bt57h\" (UID: \"166e0d16-c9f0-4993-b0de-e1cc91fd993a\") " pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.373310 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.373287    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj6nk\" (UniqueName: \"kubernetes.io/projected/036736e7-e135-4c08-be7a-8ac7d1975cdf-kube-api-access-wj6nk\") pod \"multus-additional-cni-plugins-k6nrz\" (UID: \"036736e7-e135-4c08-be7a-8ac7d1975cdf\") " pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.373310 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.373304    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96s5\" (UniqueName: \"kubernetes.io/projected/0062e4c2-a9d5-4631-b2f8-a5e223b85264-kube-api-access-n96s5\") pod \"aws-ebs-csi-driver-node-kq7s4\" (UID: \"0062e4c2-a9d5-4631-b2f8-a5e223b85264\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.373919 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.373901    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2x6\" (UniqueName: \"kubernetes.io/projected/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-kube-api-access-pm2x6\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:11:57.374822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.374798    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rrnp\" (UniqueName: \"kubernetes.io/projected/6806bb53-304b-41cf-b24c-79e2cbc4bf4d-kube-api-access-4rrnp\") pod \"ovnkube-node-vwkth\" (UID: \"6806bb53-304b-41cf-b24c-79e2cbc4bf4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.464622 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464589    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-sysconfig\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.464622 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464619    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-sys\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.464822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464636    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-sysctl-conf\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.464822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464652    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea4887a5-60be-4a3f-9580-61202226536e-etc-tuned\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.464822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464708    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-sysconfig\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.464822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464717    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-host\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.464822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464716    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-sys\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.464822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464774    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfkm\" (UniqueName: \"kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm\") pod \"network-check-target-qqdd4\" (UID: \"df43e33a-d9bd-4d2e-9a95-f7487dd3c442\") " pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:11:57.464822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464801    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-sysctl-conf\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.464822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464813    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-kubelet-config\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:57.464822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464769    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-host\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464847    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-dbus\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464886    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-kubelet-config\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464939    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464963    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-dbus\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.464967    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-var-lib-kubelet\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465001    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-sysctl-d\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465026    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-run\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465047    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-var-lib-kubelet\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465053    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzw4d\" (UniqueName: \"kubernetes.io/projected/ea4887a5-60be-4a3f-9580-61202226536e-kube-api-access-rzw4d\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.465060    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465079    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-kubernetes\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465088    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-run\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465103    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-lib-modules\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.465120    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret podName:6c8fc4b5-4d66-4ed8-948b-a9a57931649e nodeName:}" failed. No retries permitted until 2026-07-25 01:11:57.96510141 +0000 UTC m=+2.074910688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret") pod "global-pull-secret-syncer-hkv52" (UID: "6c8fc4b5-4d66-4ed8-948b-a9a57931649e") : object "kube-system"/"original-pull-secret" not registered
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465127    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-sysctl-d\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465145    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea4887a5-60be-4a3f-9580-61202226536e-tmp\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465325 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465177    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-systemd\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465960 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465205    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-lib-modules\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465960 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465212    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-modprobe-d\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465960 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465243    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-kubernetes\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465960 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465277    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-systemd\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.465960 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.465378    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea4887a5-60be-4a3f-9580-61202226536e-etc-modprobe-d\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.466899 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.466884    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea4887a5-60be-4a3f-9580-61202226536e-etc-tuned\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.467229 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.467212    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea4887a5-60be-4a3f-9580-61202226536e-tmp\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.471125 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.471102    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 25 01:11:57.471125 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.471121    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 25 01:11:57.471125 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.471131    2572 projected.go:196] Error preparing data for projected volume kube-api-access-jrfkm for pod openshift-network-diagnostics/network-check-target-qqdd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:11:57.471281 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.471185    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm podName:df43e33a-d9bd-4d2e-9a95-f7487dd3c442 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:57.971172698 +0000 UTC m=+2.080981962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jrfkm" (UniqueName: "kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm") pod "network-check-target-qqdd4" (UID: "df43e33a-d9bd-4d2e-9a95-f7487dd3c442") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:11:57.473113 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.473094    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzw4d\" (UniqueName: \"kubernetes.io/projected/ea4887a5-60be-4a3f-9580-61202226536e-kube-api-access-rzw4d\") pod \"tuned-mrk86\" (UID: \"ea4887a5-60be-4a3f-9580-61202226536e\") " pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.580897 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.580816    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:11:57.589143 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.589121    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4"
Jul 25 01:11:57.590012 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:57.589989    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9d7e4de_3c40_43fd_a2c1_67117b28520e.slice/crio-c3d2055be9e067d609c2bcb2bd1ee2f299b8f6fe42fc6076961785d82947477c WatchSource:0}: Error finding container c3d2055be9e067d609c2bcb2bd1ee2f299b8f6fe42fc6076961785d82947477c: Status 404 returned error can't find the container with id c3d2055be9e067d609c2bcb2bd1ee2f299b8f6fe42fc6076961785d82947477c
Jul 25 01:11:57.597238 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:57.597214    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0062e4c2_a9d5_4631_b2f8_a5e223b85264.slice/crio-422298ed794faa8757a14747acc9dee88a4bd416456f2953d3ca5fb0ecc41405 WatchSource:0}: Error finding container 422298ed794faa8757a14747acc9dee88a4bd416456f2953d3ca5fb0ecc41405: Status 404 returned error can't find the container with id 422298ed794faa8757a14747acc9dee88a4bd416456f2953d3ca5fb0ecc41405
Jul 25 01:11:57.598636 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.598620    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 25 01:11:57.600453 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.600436    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9x2v2"
Jul 25 01:11:57.603623 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.603605    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:11:57.608146 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:57.608122    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5d45f7_545f_4139_a706_dc19d908743f.slice/crio-8b7c16730841164f10888b203676aaceaf6e098830510fd9be036d4fbf672cdb WatchSource:0}: Error finding container 8b7c16730841164f10888b203676aaceaf6e098830510fd9be036d4fbf672cdb: Status 404 returned error can't find the container with id 8b7c16730841164f10888b203676aaceaf6e098830510fd9be036d4fbf672cdb
Jul 25 01:11:57.609327 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.609306    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fwsm2"
Jul 25 01:11:57.612349 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:57.612329    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6806bb53_304b_41cf_b24c_79e2cbc4bf4d.slice/crio-ad202b3be0796b77bd670fcf0436db866a06d5dbb1c0f7dc0c2360292ffe6564 WatchSource:0}: Error finding container ad202b3be0796b77bd670fcf0436db866a06d5dbb1c0f7dc0c2360292ffe6564: Status 404 returned error can't find the container with id ad202b3be0796b77bd670fcf0436db866a06d5dbb1c0f7dc0c2360292ffe6564
Jul 25 01:11:57.615633 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.615613    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k6nrz"
Jul 25 01:11:57.617231 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:57.617210    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0889cc33_fc59_4e79_a109_6e98bf53cc58.slice/crio-93bbfef26cc53d652c6e21a675f7bb97b192815dbe4566a228d7ac8df8e4d787 WatchSource:0}: Error finding container 93bbfef26cc53d652c6e21a675f7bb97b192815dbe4566a228d7ac8df8e4d787: Status 404 returned error can't find the container with id 93bbfef26cc53d652c6e21a675f7bb97b192815dbe4566a228d7ac8df8e4d787
Jul 25 01:11:57.621057 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.621040    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jd9wg"
Jul 25 01:11:57.624139 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:57.624118    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036736e7_e135_4c08_be7a_8ac7d1975cdf.slice/crio-59670d250a23119e1663ca1cf7fb8aa85aadbf3541620b7e0df330b5049fd52c WatchSource:0}: Error finding container 59670d250a23119e1663ca1cf7fb8aa85aadbf3541620b7e0df330b5049fd52c: Status 404 returned error can't find the container with id 59670d250a23119e1663ca1cf7fb8aa85aadbf3541620b7e0df330b5049fd52c
Jul 25 01:11:57.627507 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.627487    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bt57h"
Jul 25 01:11:57.628766 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:57.628745    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373f9d31_a96b_4a50_b460_0716a102570f.slice/crio-b886c38dcae368582e12f3227cfba705ec8845a44f20f080c2260aaf5c6587a2 WatchSource:0}: Error finding container b886c38dcae368582e12f3227cfba705ec8845a44f20f080c2260aaf5c6587a2: Status 404 returned error can't find the container with id b886c38dcae368582e12f3227cfba705ec8845a44f20f080c2260aaf5c6587a2
Jul 25 01:11:57.632857 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.632837    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mrk86"
Jul 25 01:11:57.638318 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:57.638294    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166e0d16_c9f0_4993_b0de_e1cc91fd993a.slice/crio-f7c0a3049f6f497d0905da7e91c77d592c40f500cd2a4a3fd6ce783d44cce4e0 WatchSource:0}: Error finding container f7c0a3049f6f497d0905da7e91c77d592c40f500cd2a4a3fd6ce783d44cce4e0: Status 404 returned error can't find the container with id f7c0a3049f6f497d0905da7e91c77d592c40f500cd2a4a3fd6ce783d44cce4e0
Jul 25 01:11:57.641050 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:11:57.641029    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4887a5_60be_4a3f_9580_61202226536e.slice/crio-2533e25e78c70934092a0714ea4e382890c786a6a8a209e1f693a042d8f26ae2 WatchSource:0}: Error finding container 2533e25e78c70934092a0714ea4e382890c786a6a8a209e1f693a042d8f26ae2: Status 404 returned error can't find the container with id 2533e25e78c70934092a0714ea4e382890c786a6a8a209e1f693a042d8f26ae2
Jul 25 01:11:57.869876 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.869218    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:11:57.869876 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.869393    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:11:57.869876 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.869455    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs podName:1f0a939b-732b-4313-b0f9-2ce5feb9cc5c nodeName:}" failed. No retries permitted until 2026-07-25 01:11:58.869436991 +0000 UTC m=+2.979246284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs") pod "network-metrics-daemon-nmt2v" (UID: "1f0a939b-732b-4313-b0f9-2ce5feb9cc5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:11:57.970278 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:57.970243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:57.970468 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.970436    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 25 01:11:57.970552 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:57.970502    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret podName:6c8fc4b5-4d66-4ed8-948b-a9a57931649e nodeName:}" failed. No retries permitted until 2026-07-25 01:11:58.970481848 +0000 UTC m=+3.080291115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret") pod "global-pull-secret-syncer-hkv52" (UID: "6c8fc4b5-4d66-4ed8-948b-a9a57931649e") : object "kube-system"/"original-pull-secret" not registered
Jul 25 01:11:58.071090 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.071053    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfkm\" (UniqueName: \"kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm\") pod \"network-check-target-qqdd4\" (UID: \"df43e33a-d9bd-4d2e-9a95-f7487dd3c442\") " pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:11:58.071275 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:58.071252    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 25 01:11:58.071275 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:58.071273    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 25 01:11:58.071391 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:58.071293    2572 projected.go:196] Error preparing data for projected volume kube-api-access-jrfkm for pod openshift-network-diagnostics/network-check-target-qqdd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:11:58.071391 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:58.071354    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm podName:df43e33a-d9bd-4d2e-9a95-f7487dd3c442 nodeName:}" failed. No retries permitted until 2026-07-25 01:11:59.071334997 +0000 UTC m=+3.181144264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrfkm" (UniqueName: "kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm") pod "network-check-target-qqdd4" (UID: "df43e33a-d9bd-4d2e-9a95-f7487dd3c442") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:11:58.290235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.290153    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-24 01:06:57 +0000 UTC" deadline="2028-01-29 16:32:40.092184445 +0000 UTC"
Jul 25 01:11:58.290235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.290232    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13287h20m41.801956751s"
Jul 25 01:11:58.361183 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.361152    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:11:58.361342 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:58.361287    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:11:58.370852 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.370813    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrk86" event={"ID":"ea4887a5-60be-4a3f-9580-61202226536e","Type":"ContainerStarted","Data":"2533e25e78c70934092a0714ea4e382890c786a6a8a209e1f693a042d8f26ae2"}
Jul 25 01:11:58.378469 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.378411    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bt57h" event={"ID":"166e0d16-c9f0-4993-b0de-e1cc91fd993a","Type":"ContainerStarted","Data":"f7c0a3049f6f497d0905da7e91c77d592c40f500cd2a4a3fd6ce783d44cce4e0"}
Jul 25 01:11:58.387008 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.386977    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6nrz" event={"ID":"036736e7-e135-4c08-be7a-8ac7d1975cdf","Type":"ContainerStarted","Data":"59670d250a23119e1663ca1cf7fb8aa85aadbf3541620b7e0df330b5049fd52c"}
Jul 25 01:11:58.395118 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.395042    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth" event={"ID":"6806bb53-304b-41cf-b24c-79e2cbc4bf4d","Type":"ContainerStarted","Data":"ad202b3be0796b77bd670fcf0436db866a06d5dbb1c0f7dc0c2360292ffe6564"}
Jul 25 01:11:58.398619 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.398594    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jd9wg" event={"ID":"373f9d31-a96b-4a50-b460-0716a102570f","Type":"ContainerStarted","Data":"b886c38dcae368582e12f3227cfba705ec8845a44f20f080c2260aaf5c6587a2"}
Jul 25 01:11:58.416119 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.415946    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fwsm2" event={"ID":"0889cc33-fc59-4e79-a109-6e98bf53cc58","Type":"ContainerStarted","Data":"93bbfef26cc53d652c6e21a675f7bb97b192815dbe4566a228d7ac8df8e4d787"}
Jul 25 01:11:58.423559 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.423530    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9x2v2" event={"ID":"2c5d45f7-545f-4139-a706-dc19d908743f","Type":"ContainerStarted","Data":"8b7c16730841164f10888b203676aaceaf6e098830510fd9be036d4fbf672cdb"}
Jul 25 01:11:58.450649 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.450564    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4" event={"ID":"0062e4c2-a9d5-4631-b2f8-a5e223b85264","Type":"ContainerStarted","Data":"422298ed794faa8757a14747acc9dee88a4bd416456f2953d3ca5fb0ecc41405"}
Jul 25 01:11:58.458922 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.458847    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rk9b7" event={"ID":"d9d7e4de-3c40-43fd-a2c1-67117b28520e","Type":"ContainerStarted","Data":"c3d2055be9e067d609c2bcb2bd1ee2f299b8f6fe42fc6076961785d82947477c"}
Jul 25 01:11:58.657626 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.657593    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 25 01:11:58.881956 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.881916    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:11:58.882142 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:58.882079    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:11:58.882186 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:58.882151    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs podName:1f0a939b-732b-4313-b0f9-2ce5feb9cc5c nodeName:}" failed. No retries permitted until 2026-07-25 01:12:00.882129044 +0000 UTC m=+4.991938351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs") pod "network-metrics-daemon-nmt2v" (UID: "1f0a939b-732b-4313-b0f9-2ce5feb9cc5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:11:58.984049 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:58.983959    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:58.984210 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:58.984177    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 25 01:11:58.984329 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:58.984244    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret podName:6c8fc4b5-4d66-4ed8-948b-a9a57931649e nodeName:}" failed. No retries permitted until 2026-07-25 01:12:00.984224881 +0000 UTC m=+5.094034147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret") pod "global-pull-secret-syncer-hkv52" (UID: "6c8fc4b5-4d66-4ed8-948b-a9a57931649e") : object "kube-system"/"original-pull-secret" not registered
Jul 25 01:11:59.084928 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:59.084755    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfkm\" (UniqueName: \"kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm\") pod \"network-check-target-qqdd4\" (UID: \"df43e33a-d9bd-4d2e-9a95-f7487dd3c442\") " pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:11:59.085107 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:59.085000    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 25 01:11:59.085107 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:59.085024    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 25 01:11:59.085107 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:59.085036    2572 projected.go:196] Error preparing data for projected volume kube-api-access-jrfkm for pod openshift-network-diagnostics/network-check-target-qqdd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:11:59.085107 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:59.085098    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm podName:df43e33a-d9bd-4d2e-9a95-f7487dd3c442 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:01.085079522 +0000 UTC m=+5.194888785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrfkm" (UniqueName: "kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm") pod "network-check-target-qqdd4" (UID: "df43e33a-d9bd-4d2e-9a95-f7487dd3c442") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:11:59.291038 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:59.290920    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-24 01:06:57 +0000 UTC" deadline="2028-03-06 16:08:48.640556797 +0000 UTC"
Jul 25 01:11:59.291038 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:59.290983    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14174h56m49.349577764s"
Jul 25 01:11:59.348459 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:59.347908    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:11:59.348459 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:59.348106    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:11:59.348693 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:59.348589    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:11:59.348756 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:11:59.348733    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:11:59.855646 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:11:59.855611    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 25 01:12:00.350926 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:00.350026    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:00.350926 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:00.350156    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:00.902363 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:00.902320    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:00.902566 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:00.902472    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:12:00.902566 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:00.902534    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs podName:1f0a939b-732b-4313-b0f9-2ce5feb9cc5c nodeName:}" failed. No retries permitted until 2026-07-25 01:12:04.902515562 +0000 UTC m=+9.012324840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs") pod "network-metrics-daemon-nmt2v" (UID: "1f0a939b-732b-4313-b0f9-2ce5feb9cc5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:12:01.003074 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:01.003035    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:01.003262 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:01.003231    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 25 01:12:01.003324 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:01.003294    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret podName:6c8fc4b5-4d66-4ed8-948b-a9a57931649e nodeName:}" failed. No retries permitted until 2026-07-25 01:12:05.003275437 +0000 UTC m=+9.113084727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret") pod "global-pull-secret-syncer-hkv52" (UID: "6c8fc4b5-4d66-4ed8-948b-a9a57931649e") : object "kube-system"/"original-pull-secret" not registered
Jul 25 01:12:01.104294 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:01.103639    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfkm\" (UniqueName: \"kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm\") pod \"network-check-target-qqdd4\" (UID: \"df43e33a-d9bd-4d2e-9a95-f7487dd3c442\") " pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:01.104294 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:01.103811    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 25 01:12:01.104294 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:01.103833    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 25 01:12:01.104294 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:01.103846    2572 projected.go:196] Error preparing data for projected volume kube-api-access-jrfkm for pod openshift-network-diagnostics/network-check-target-qqdd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:12:01.104294 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:01.103904    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm podName:df43e33a-d9bd-4d2e-9a95-f7487dd3c442 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:05.103885928 +0000 UTC m=+9.213695194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrfkm" (UniqueName: "kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm") pod "network-check-target-qqdd4" (UID: "df43e33a-d9bd-4d2e-9a95-f7487dd3c442") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:12:01.348364 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:01.347858    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:01.348364 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:01.348001    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:01.348364 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:01.348100    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:01.348364 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:01.348231    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:02.347883 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:02.347840    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:02.348396 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:02.347991    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:03.347480 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:03.347449    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:03.347665 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:03.347585    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:03.347976 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:03.347954    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:03.348363 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:03.348062    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:04.348241 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:04.348198    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:04.348669 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:04.348347    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:04.937396 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:04.937357    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:04.937588 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:04.937525    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:12:04.937653 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:04.937600    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs podName:1f0a939b-732b-4313-b0f9-2ce5feb9cc5c nodeName:}" failed. No retries permitted until 2026-07-25 01:12:12.937580403 +0000 UTC m=+17.047389683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs") pod "network-metrics-daemon-nmt2v" (UID: "1f0a939b-732b-4313-b0f9-2ce5feb9cc5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:12:05.038518 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:05.038487    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:05.038702 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:05.038602    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 25 01:12:05.038702 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:05.038650    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret podName:6c8fc4b5-4d66-4ed8-948b-a9a57931649e nodeName:}" failed. No retries permitted until 2026-07-25 01:12:13.038637346 +0000 UTC m=+17.148446607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret") pod "global-pull-secret-syncer-hkv52" (UID: "6c8fc4b5-4d66-4ed8-948b-a9a57931649e") : object "kube-system"/"original-pull-secret" not registered
Jul 25 01:12:05.139404 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:05.139366    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfkm\" (UniqueName: \"kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm\") pod \"network-check-target-qqdd4\" (UID: \"df43e33a-d9bd-4d2e-9a95-f7487dd3c442\") " pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:05.139571 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:05.139547    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 25 01:12:05.139653 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:05.139575    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 25 01:12:05.139653 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:05.139589    2572 projected.go:196] Error preparing data for projected volume kube-api-access-jrfkm for pod openshift-network-diagnostics/network-check-target-qqdd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:12:05.139749 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:05.139656    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm podName:df43e33a-d9bd-4d2e-9a95-f7487dd3c442 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:13.13963489 +0000 UTC m=+17.249444166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrfkm" (UniqueName: "kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm") pod "network-check-target-qqdd4" (UID: "df43e33a-d9bd-4d2e-9a95-f7487dd3c442") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:12:05.347966 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:05.347881    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:05.348122 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:05.347881    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:05.348122 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:05.348024    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:05.348235 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:05.348126    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:06.348665 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:06.348632    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:06.349134 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:06.348764    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:07.347248 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:07.347184    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:07.347446 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:07.347316    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:07.347548 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:07.347184    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:07.347684 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:07.347654    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:08.347712 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:08.347679    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:08.348095 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:08.347778    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:09.347851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:09.347817    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:09.348277 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:09.347928    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:09.348277 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:09.347953    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:09.348277 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:09.347997    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:10.347655 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:10.347622    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:10.347837 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:10.347752    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:11.347447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:11.347414    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:11.347831 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:11.347415    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:11.347831 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:11.347548    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:11.347831 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:11.347628    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:12.347109 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:12.347068    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:12.347266 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:12.347186    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:12.993402 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:12.993365    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:12.993888 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:12.993541    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:12:12.993888 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:12.993621    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs podName:1f0a939b-732b-4313-b0f9-2ce5feb9cc5c nodeName:}" failed. No retries permitted until 2026-07-25 01:12:28.993601226 +0000 UTC m=+33.103410505 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs") pod "network-metrics-daemon-nmt2v" (UID: "1f0a939b-732b-4313-b0f9-2ce5feb9cc5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:12:13.094474 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:13.094435    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:13.094662 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:13.094573    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 25 01:12:13.094662 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:13.094640    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret podName:6c8fc4b5-4d66-4ed8-948b-a9a57931649e nodeName:}" failed. No retries permitted until 2026-07-25 01:12:29.094622566 +0000 UTC m=+33.204431847 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret") pod "global-pull-secret-syncer-hkv52" (UID: "6c8fc4b5-4d66-4ed8-948b-a9a57931649e") : object "kube-system"/"original-pull-secret" not registered
Jul 25 01:12:13.195591 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:13.195555    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfkm\" (UniqueName: \"kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm\") pod \"network-check-target-qqdd4\" (UID: \"df43e33a-d9bd-4d2e-9a95-f7487dd3c442\") " pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:13.195755 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:13.195732    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 25 01:12:13.195802 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:13.195762    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 25 01:12:13.195802 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:13.195771    2572 projected.go:196] Error preparing data for projected volume kube-api-access-jrfkm for pod openshift-network-diagnostics/network-check-target-qqdd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:12:13.195870 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:13.195820    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm podName:df43e33a-d9bd-4d2e-9a95-f7487dd3c442 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:29.195806526 +0000 UTC m=+33.305615788 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrfkm" (UniqueName: "kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm") pod "network-check-target-qqdd4" (UID: "df43e33a-d9bd-4d2e-9a95-f7487dd3c442") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:12:13.347481 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:13.347410    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:13.347620 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:13.347410    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:13.347620 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:13.347542    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:13.347738 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:13.347638    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:14.347256 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:14.347195    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:14.347734 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:14.347339    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:15.346960 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:15.346920    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:15.347164 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:15.347066    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:15.347164 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:15.347125    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:15.347271 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:15.347239    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:16.347610 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:16.347578    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:16.348052 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:16.347683    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:17.347285 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:17.347261    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:17.347414 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:17.347365    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:17.347451 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:17.347426    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:17.347539 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:17.347521    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:18.347711 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.347404    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:18.348199 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:18.347795    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:18.502089 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.502063    2572 generic.go:356] "Generic (PLEG): container finished" podID="0dc2e8379747207d092a55285ea986ce" containerID="2588e2342e1d030c785eb617d4eb1cb1d121eadf3fdeec76b1cceb151361aaa7" exitCode=0
Jul 25 01:12:18.502182 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.502142    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal" event={"ID":"0dc2e8379747207d092a55285ea986ce","Type":"ContainerDied","Data":"2588e2342e1d030c785eb617d4eb1cb1d121eadf3fdeec76b1cceb151361aaa7"}
Jul 25 01:12:18.503599 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.503580    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jd9wg" event={"ID":"373f9d31-a96b-4a50-b460-0716a102570f","Type":"ContainerStarted","Data":"6dd713f3b57c87b2d64264f88d78b29fc65c8dcb3a221d0e506b56c99d82351f"}
Jul 25 01:12:18.505135 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.505113    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fwsm2" event={"ID":"0889cc33-fc59-4e79-a109-6e98bf53cc58","Type":"ContainerStarted","Data":"950999bdf934bc1d5978d189c78ec338b68dbbf77a68c48d5142223cb80ebcd0"}
Jul 25 01:12:18.506784 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.506678    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4" event={"ID":"0062e4c2-a9d5-4631-b2f8-a5e223b85264","Type":"ContainerStarted","Data":"3da7277d96dd7dd86fd7f0737cfdfed67ee07d9d8853901d46ae1d9c665372c0"}
Jul 25 01:12:18.508072 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.508056    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rk9b7" event={"ID":"d9d7e4de-3c40-43fd-a2c1-67117b28520e","Type":"ContainerStarted","Data":"cfa0d26e8a1db76064e90b92dc4afbb4718b3ac9d74908d9e91b2f683558e64f"}
Jul 25 01:12:18.509365 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.509348    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-99.ec2.internal" event={"ID":"47c77adca673d868115fa74ace1d7b3f","Type":"ContainerStarted","Data":"1dcfa189a077cf5b8c67acdfec61e2d11ad8873165d48a23bef55f841037264d"}
Jul 25 01:12:18.510688 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.510671    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mrk86" event={"ID":"ea4887a5-60be-4a3f-9580-61202226536e","Type":"ContainerStarted","Data":"650e3f884f5cf548bcf08cfd2e556118b1333722992845767ef77e1337507c15"}
Jul 25 01:12:18.511975 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.511943    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bt57h" event={"ID":"166e0d16-c9f0-4993-b0de-e1cc91fd993a","Type":"ContainerStarted","Data":"6cba164408f2c6fcbc174b31be937674772d927b85fd8289b49aca8eca9111a7"}
Jul 25 01:12:18.513442 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.513422    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6nrz" event={"ID":"036736e7-e135-4c08-be7a-8ac7d1975cdf","Type":"ContainerStarted","Data":"af6abd7c2fd2a2a9cd0ceff398a1705ab1938ae7ec94890642e13ea9c79b1db2"}
Jul 25 01:12:18.516216 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.516192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth" event={"ID":"6806bb53-304b-41cf-b24c-79e2cbc4bf4d","Type":"ContainerStarted","Data":"05f766366eb68be8fc0fd167d981a5cd5a5e48ed2bf77161191dbe3c30312c1e"}
Jul 25 01:12:18.516298 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.516269    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth" event={"ID":"6806bb53-304b-41cf-b24c-79e2cbc4bf4d","Type":"ContainerStarted","Data":"c11708447372a2b7f46ec656a57fbae7dad53d9239cf72b7fb9c4c5cec2010e2"}
Jul 25 01:12:18.516364 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.516349    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth" event={"ID":"6806bb53-304b-41cf-b24c-79e2cbc4bf4d","Type":"ContainerStarted","Data":"d4dda1815f7ee9e1edb85bb41633b81b899c9eb1e922de8bffdf3ee4b2805a98"}
Jul 25 01:12:18.516416 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.516402    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth" event={"ID":"6806bb53-304b-41cf-b24c-79e2cbc4bf4d","Type":"ContainerStarted","Data":"f3e50cb05ae44481824fa0ecf5e6bcb88cc9d11becd3bc458a8b11e7e9ad805e"}
Jul 25 01:12:18.516469 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.516454    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth" event={"ID":"6806bb53-304b-41cf-b24c-79e2cbc4bf4d","Type":"ContainerStarted","Data":"d70164d9a6e7685045d4212ef4e09b59a324615bc96b6730ca616efac54ef9dc"}
Jul 25 01:12:18.541922 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.541894    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:12:18.542051 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.541991    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rk9b7" podStartSLOduration=2.534217456 podStartE2EDuration="22.541974472s" totalImagesPullingTime="20.007757016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:18.527174413 +0000 UTC m=+22.636983697" watchObservedRunningTime="2026-07-25 01:12:18.541974472 +0000 UTC m=+22.651783753"
Jul 25 01:12:18.542465 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.542448    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:12:18.542641 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.542607    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fwsm2" podStartSLOduration=2.495803684 podStartE2EDuration="22.542597868s" totalImagesPullingTime="20.046794184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:18.542058553 +0000 UTC m=+22.651867838" watchObservedRunningTime="2026-07-25 01:12:18.542597868 +0000 UTC m=+22.652407152"
Jul 25 01:12:18.555773 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.555729    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jd9wg" podStartSLOduration=1.58691528 podStartE2EDuration="21.555714135s" totalImagesPullingTime="19.968798855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:18.555660206 +0000 UTC m=+22.665469491" watchObservedRunningTime="2026-07-25 01:12:18.555714135 +0000 UTC m=+22.665523419"
Jul 25 01:12:18.573048 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.573015    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bt57h" podStartSLOduration=1.5370079479999998 podStartE2EDuration="21.573004001s" totalImagesPullingTime="20.035996053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:18.572504549 +0000 UTC m=+22.682313833" watchObservedRunningTime="2026-07-25 01:12:18.573004001 +0000 UTC m=+22.682813285"
Jul 25 01:12:18.585521 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.585485    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-99.ec2.internal" podStartSLOduration=21.585473141 podStartE2EDuration="21.585473141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:18.585102968 +0000 UTC m=+22.694912252" watchObservedRunningTime="2026-07-25 01:12:18.585473141 +0000 UTC m=+22.695282425"
Jul 25 01:12:18.622143 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:18.622053    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mrk86" podStartSLOduration=1.597474439 podStartE2EDuration="21.622040394s" totalImagesPullingTime="20.024565955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:18.621805657 +0000 UTC m=+22.731614942" watchObservedRunningTime="2026-07-25 01:12:18.622040394 +0000 UTC m=+22.731849677"
Jul 25 01:12:19.348105 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:19.347872    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:19.348105 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:19.347931    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:19.348711 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:19.348120    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:19.348711 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:19.348187    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:19.520357 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:19.520305    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal" event={"ID":"0dc2e8379747207d092a55285ea986ce","Type":"ContainerStarted","Data":"a6972fddf2628b8e6fa88176230f85e06ea36dbf3cd116cedd56b682c8f7720b"}
Jul 25 01:12:19.521720 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:19.521695    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9x2v2" event={"ID":"2c5d45f7-545f-4139-a706-dc19d908743f","Type":"ContainerStarted","Data":"11b9fad3556a369ecd1a59c5fcbeab5bc2c3e53a1f9a67fad52cdbfae598c942"}
Jul 25 01:12:19.523086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:19.523064    2572 generic.go:356] "Generic (PLEG): container finished" podID="036736e7-e135-4c08-be7a-8ac7d1975cdf" containerID="af6abd7c2fd2a2a9cd0ceff398a1705ab1938ae7ec94890642e13ea9c79b1db2" exitCode=0
Jul 25 01:12:19.523203 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:19.523113    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6nrz" event={"ID":"036736e7-e135-4c08-be7a-8ac7d1975cdf","Type":"ContainerDied","Data":"af6abd7c2fd2a2a9cd0ceff398a1705ab1938ae7ec94890642e13ea9c79b1db2"}
Jul 25 01:12:19.525958 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:19.525933    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth" event={"ID":"6806bb53-304b-41cf-b24c-79e2cbc4bf4d","Type":"ContainerStarted","Data":"d3dc0ecb26b2c4caa02c66939d8afe7672fb6ff91e72c28aa2e516b405e2465a"}
Jul 25 01:12:19.536965 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:19.536919    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-99.ec2.internal" podStartSLOduration=22.536906875 podStartE2EDuration="22.536906875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:19.53665469 +0000 UTC m=+23.646463976" watchObservedRunningTime="2026-07-25 01:12:19.536906875 +0000 UTC m=+23.646716158"
Jul 25 01:12:19.551002 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:19.550964    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9x2v2" podStartSLOduration=3.49676182 podStartE2EDuration="23.550951179s" totalImagesPullingTime="20.054189359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:19.550791375 +0000 UTC m=+23.660600669" watchObservedRunningTime="2026-07-25 01:12:19.550951179 +0000 UTC m=+23.660760463"
Jul 25 01:12:19.723913 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:19.723892    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 25 01:12:20.316925 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:20.316823    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-25T01:12:19.723908627Z","UUID":"0ac76a8a-7cff-4d34-8918-0181fd88c4ce","Handler":null,"Name":"","Endpoint":""}
Jul 25 01:12:20.318513 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:20.318490    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 25 01:12:20.318653 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:20.318534    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 25 01:12:20.346980 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:20.346955    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:20.347130 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:20.347070    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:20.531144 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:20.531113    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 25 01:12:20.531739 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:20.531145    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4" event={"ID":"0062e4c2-a9d5-4631-b2f8-a5e223b85264","Type":"ContainerStarted","Data":"327f286e5e3296b6c43fee77b3e7234ad64ed78db9230f9acfc02a6b213cc4cb"}
Jul 25 01:12:21.348026 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:21.347949    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:21.348175 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:21.348077    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:21.348175 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:21.348137    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:21.348294 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:21.348264    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:21.536081 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:21.536044    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4" event={"ID":"0062e4c2-a9d5-4631-b2f8-a5e223b85264","Type":"ContainerStarted","Data":"5a5f1824a13d969f11699ce2d4710e03c2d4062ab9dd41c02a2cd8fedb186c81"}
Jul 25 01:12:21.539523 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:21.539498    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth" event={"ID":"6806bb53-304b-41cf-b24c-79e2cbc4bf4d","Type":"ContainerStarted","Data":"d13973c2a978b2c9f1ec40b500502b4d25719f8ca0cb90d4d1e5beea944d8be4"}
Jul 25 01:12:21.554294 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:21.554253    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kq7s4" podStartSLOduration=3.180333177 podStartE2EDuration="25.554239376s" totalImagesPullingTime="22.373906199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:21.553652919 +0000 UTC m=+25.663462202" watchObservedRunningTime="2026-07-25 01:12:21.554239376 +0000 UTC m=+25.664048661"
Jul 25 01:12:22.347510 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:22.347477    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:22.347717 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:22.347606    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:22.959927 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:22.959895    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:12:22.960425 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:22.960045    2572 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 25 01:12:22.960501 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:22.960463    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rk9b7"
Jul 25 01:12:23.347504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:23.347467    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:23.347504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:23.347502    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:23.347738 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:23.347594    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:23.347738 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:23.347712    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:24.347656 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:24.347505    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:24.348280 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:24.347957    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:24.547193 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:24.547160    2572 generic.go:356] "Generic (PLEG): container finished" podID="036736e7-e135-4c08-be7a-8ac7d1975cdf" containerID="662c35f74853c97c3aa93369cdf691e6070bb427a3a4900a24a7e23ad9febd43" exitCode=0
Jul 25 01:12:24.547398 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:24.547247    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6nrz" event={"ID":"036736e7-e135-4c08-be7a-8ac7d1975cdf","Type":"ContainerDied","Data":"662c35f74853c97c3aa93369cdf691e6070bb427a3a4900a24a7e23ad9febd43"}
Jul 25 01:12:24.550823 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:24.550797    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth" event={"ID":"6806bb53-304b-41cf-b24c-79e2cbc4bf4d","Type":"ContainerStarted","Data":"eaaa54c2c87d7460ebb8bfcf20870f5afc3fd257c472973839b0bdbba1913788"}
Jul 25 01:12:24.551063 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:24.551026    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:12:24.551158 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:24.551113    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:12:24.551259 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:24.551162    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:12:24.565936 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:24.565912    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:12:24.566183 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:24.566159    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:12:24.590880 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:24.590847    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth" podStartSLOduration=8.237515794 podStartE2EDuration="28.590835065s" totalImagesPullingTime="20.353319271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:24.590624078 +0000 UTC m=+28.700433361" watchObservedRunningTime="2026-07-25 01:12:24.590835065 +0000 UTC m=+28.700644348"
Jul 25 01:12:25.347750 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:25.347720    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:25.348370 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:25.347723    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:25.348370 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:25.347826    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:25.348370 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:25.347916    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:25.554946 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:25.554921    2572 generic.go:356] "Generic (PLEG): container finished" podID="036736e7-e135-4c08-be7a-8ac7d1975cdf" containerID="6d80663d9f180ed6db379dc20bb99abc9087dc903e8105216b074bf9cdfd3326" exitCode=0
Jul 25 01:12:25.555072 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:25.555000    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6nrz" event={"ID":"036736e7-e135-4c08-be7a-8ac7d1975cdf","Type":"ContainerDied","Data":"6d80663d9f180ed6db379dc20bb99abc9087dc903e8105216b074bf9cdfd3326"}
Jul 25 01:12:25.846607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:25.846418    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qqdd4"]
Jul 25 01:12:25.846752 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:25.846704    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:25.846827 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:25.846806    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:25.847007 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:25.846986    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hkv52"]
Jul 25 01:12:25.847092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:25.847073    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:25.847209 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:25.847183    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:25.849162 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:25.849140    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmt2v"]
Jul 25 01:12:25.849246 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:25.849231    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:25.849384 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:25.849363    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:26.559679 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:26.559645    2572 generic.go:356] "Generic (PLEG): container finished" podID="036736e7-e135-4c08-be7a-8ac7d1975cdf" containerID="f49713dc85d02d111675c859a7adbd631c2a786a534b1a5d9078aed39e5bc00c" exitCode=0
Jul 25 01:12:26.560009 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:26.559723    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6nrz" event={"ID":"036736e7-e135-4c08-be7a-8ac7d1975cdf","Type":"ContainerDied","Data":"f49713dc85d02d111675c859a7adbd631c2a786a534b1a5d9078aed39e5bc00c"}
Jul 25 01:12:27.346992 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:27.346961    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:27.347141 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:27.347009    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:27.347141 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:27.347009    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:27.347141 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:27.347092    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:27.347285 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:27.347201    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:27.347285 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:27.347269    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:29.016536 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:29.016501    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:29.017005 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:29.016684    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:12:29.017005 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:29.016765    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs podName:1f0a939b-732b-4313-b0f9-2ce5feb9cc5c nodeName:}" failed. No retries permitted until 2026-07-25 01:13:01.016743191 +0000 UTC m=+65.126552454 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs") pod "network-metrics-daemon-nmt2v" (UID: "1f0a939b-732b-4313-b0f9-2ce5feb9cc5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 25 01:12:29.117381 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:29.117345    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:29.117550 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:29.117499    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 25 01:12:29.117611 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:29.117571    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret podName:6c8fc4b5-4d66-4ed8-948b-a9a57931649e nodeName:}" failed. No retries permitted until 2026-07-25 01:13:01.117548644 +0000 UTC m=+65.227357906 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret") pod "global-pull-secret-syncer-hkv52" (UID: "6c8fc4b5-4d66-4ed8-948b-a9a57931649e") : object "kube-system"/"original-pull-secret" not registered
Jul 25 01:12:29.218468 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:29.218435    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfkm\" (UniqueName: \"kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm\") pod \"network-check-target-qqdd4\" (UID: \"df43e33a-d9bd-4d2e-9a95-f7487dd3c442\") " pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:29.218661 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:29.218638    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 25 01:12:29.218714 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:29.218668    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 25 01:12:29.218714 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:29.218681    2572 projected.go:196] Error preparing data for projected volume kube-api-access-jrfkm for pod openshift-network-diagnostics/network-check-target-qqdd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:12:29.218802 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:29.218743    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm podName:df43e33a-d9bd-4d2e-9a95-f7487dd3c442 nodeName:}" failed. No retries permitted until 2026-07-25 01:13:01.218723865 +0000 UTC m=+65.328533142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrfkm" (UniqueName: "kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm") pod "network-check-target-qqdd4" (UID: "df43e33a-d9bd-4d2e-9a95-f7487dd3c442") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 25 01:12:29.347278 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:29.347195    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:29.347278 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:29.347266    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:29.347506 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:29.347408    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:29.347506 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:29.347421    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nmt2v" podUID="1f0a939b-732b-4313-b0f9-2ce5feb9cc5c"
Jul 25 01:12:29.347609 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:29.347500    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qqdd4" podUID="df43e33a-d9bd-4d2e-9a95-f7487dd3c442"
Jul 25 01:12:29.347609 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:29.347583    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hkv52" podUID="6c8fc4b5-4d66-4ed8-948b-a9a57931649e"
Jul 25 01:12:30.773846 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.773816    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-99.ec2.internal" event="NodeReady"
Jul 25 01:12:30.774307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.773945    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 25 01:12:30.822497 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.822471    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2bkbl"]
Jul 25 01:12:30.837557 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.837530    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x62hp"]
Jul 25 01:12:30.837670 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.837645    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:30.840239 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.840213    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fxvm2\""
Jul 25 01:12:30.840239 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.840232    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 25 01:12:30.840432 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.840273    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 25 01:12:30.851734 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.851717    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f58d88f64-52gnj"]
Jul 25 01:12:30.851836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.851821    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x62hp"
Jul 25 01:12:30.854401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.854385    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-krsnh\""
Jul 25 01:12:30.854553 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.854542    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 25 01:12:30.854680 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.854667    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 25 01:12:30.855036 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.855010    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 25 01:12:30.874033 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.874015    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x62hp"]
Jul 25 01:12:30.874134 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.874042    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2bkbl"]
Jul 25 01:12:30.874134 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.874058    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f58d88f64-52gnj"]
Jul 25 01:12:30.874197 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.874140    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:30.876589 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.876570    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 25 01:12:30.876668 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.876589    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g77zs\""
Jul 25 01:12:30.876711 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.876676    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 25 01:12:30.877096 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.877081    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 25 01:12:30.881379 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.881360    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 25 01:12:30.930623 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.930598    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183e983e-4d28-47d2-be8e-36cfd1293dae-tmp-dir\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:30.930780 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.930630    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdbw9\" (UniqueName: \"kubernetes.io/projected/cc0d00f8-2138-4cf2-98bf-d1898e41ddd1-kube-api-access-mdbw9\") pod \"ingress-canary-x62hp\" (UID: \"cc0d00f8-2138-4cf2-98bf-d1898e41ddd1\") " pod="openshift-ingress-canary/ingress-canary-x62hp"
Jul 25 01:12:30.930780 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.930663    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc0d00f8-2138-4cf2-98bf-d1898e41ddd1-cert\") pod \"ingress-canary-x62hp\" (UID: \"cc0d00f8-2138-4cf2-98bf-d1898e41ddd1\") " pod="openshift-ingress-canary/ingress-canary-x62hp"
Jul 25 01:12:30.930780 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.930755    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/183e983e-4d28-47d2-be8e-36cfd1293dae-config-volume\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:30.930926 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.930792    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:30.930926 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:30.930824    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c57c5\" (UniqueName: \"kubernetes.io/projected/183e983e-4d28-47d2-be8e-36cfd1293dae-kube-api-access-c57c5\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:31.031766 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.031711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.031879 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.031782    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183e983e-4d28-47d2-be8e-36cfd1293dae-tmp-dir\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:31.031879 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.031813    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdbw9\" (UniqueName: \"kubernetes.io/projected/cc0d00f8-2138-4cf2-98bf-d1898e41ddd1-kube-api-access-mdbw9\") pod \"ingress-canary-x62hp\" (UID: \"cc0d00f8-2138-4cf2-98bf-d1898e41ddd1\") " pod="openshift-ingress-canary/ingress-canary-x62hp"
Jul 25 01:12:31.031879 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.031837    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edddd9d4-f442-4543-ae8b-f73319f9fbd4-ca-trust-extracted\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.031879 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.031858    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-certificates\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.032050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.031907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-image-registry-private-configuration\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.032050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.031956    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc0d00f8-2138-4cf2-98bf-d1898e41ddd1-cert\") pod \"ingress-canary-x62hp\" (UID: \"cc0d00f8-2138-4cf2-98bf-d1898e41ddd1\") " pod="openshift-ingress-canary/ingress-canary-x62hp"
Jul 25 01:12:31.032050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.031979    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-installation-pull-secrets\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.032050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.032014    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-trusted-ca\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.032050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.032037    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ghqz\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-kube-api-access-6ghqz\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.032269 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.032058    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183e983e-4d28-47d2-be8e-36cfd1293dae-tmp-dir\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:31.032269 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.032094    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/183e983e-4d28-47d2-be8e-36cfd1293dae-config-volume\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:31.032269 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.032124    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-bound-sa-token\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.032269 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.032150    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:31.032269 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.032178    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c57c5\" (UniqueName: \"kubernetes.io/projected/183e983e-4d28-47d2-be8e-36cfd1293dae-kube-api-access-c57c5\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:31.032269 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:31.032223    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 25 01:12:31.032576 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:31.032306    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls podName:183e983e-4d28-47d2-be8e-36cfd1293dae nodeName:}" failed. No retries permitted until 2026-07-25 01:12:31.532288584 +0000 UTC m=+35.642097846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls") pod "dns-default-2bkbl" (UID: "183e983e-4d28-47d2-be8e-36cfd1293dae") : secret "dns-default-metrics-tls" not found
Jul 25 01:12:31.032673 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.032654    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/183e983e-4d28-47d2-be8e-36cfd1293dae-config-volume\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:31.036233 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.036207    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc0d00f8-2138-4cf2-98bf-d1898e41ddd1-cert\") pod \"ingress-canary-x62hp\" (UID: \"cc0d00f8-2138-4cf2-98bf-d1898e41ddd1\") " pod="openshift-ingress-canary/ingress-canary-x62hp"
Jul 25 01:12:31.042084 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.042066    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdbw9\" (UniqueName: \"kubernetes.io/projected/cc0d00f8-2138-4cf2-98bf-d1898e41ddd1-kube-api-access-mdbw9\") pod \"ingress-canary-x62hp\" (UID: \"cc0d00f8-2138-4cf2-98bf-d1898e41ddd1\") " pod="openshift-ingress-canary/ingress-canary-x62hp"
Jul 25 01:12:31.042469 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.042439    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c57c5\" (UniqueName: \"kubernetes.io/projected/183e983e-4d28-47d2-be8e-36cfd1293dae-kube-api-access-c57c5\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:31.132558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.132428    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.132664 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:31.132574    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 25 01:12:31.132664 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:31.132592    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f58d88f64-52gnj: secret "image-registry-tls" not found
Jul 25 01:12:31.132664 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.132597    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edddd9d4-f442-4543-ae8b-f73319f9fbd4-ca-trust-extracted\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.132664 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.132620    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-certificates\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.132664 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:31.132637    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls podName:edddd9d4-f442-4543-ae8b-f73319f9fbd4 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:31.632623292 +0000 UTC m=+35.742432553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls") pod "image-registry-6f58d88f64-52gnj" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4") : secret "image-registry-tls" not found
Jul 25 01:12:31.132941 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.132669    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-image-registry-private-configuration\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.132941 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.132706    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-installation-pull-secrets\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.132941 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.132732    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-trusted-ca\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.132941 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.132759    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ghqz\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-kube-api-access-6ghqz\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.132941 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.132792    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-bound-sa-token\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.133168 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.132961    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edddd9d4-f442-4543-ae8b-f73319f9fbd4-ca-trust-extracted\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.133226 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.133186    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-certificates\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.133665 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.133647    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-trusted-ca\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.135132 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.135107    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-installation-pull-secrets\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.135218 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.135160    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-image-registry-private-configuration\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.141424 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.141402    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ghqz\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-kube-api-access-6ghqz\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.141529 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.141444    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-bound-sa-token\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.162641 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.162615    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x62hp"
Jul 25 01:12:31.314009 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.313943    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x62hp"]
Jul 25 01:12:31.317533 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:12:31.317511    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0d00f8_2138_4cf2_98bf_d1898e41ddd1.slice/crio-390975a18f15f0846d490b7b711bd0a9298fe67d62626e789eecb38dbf7f103e WatchSource:0}: Error finding container 390975a18f15f0846d490b7b711bd0a9298fe67d62626e789eecb38dbf7f103e: Status 404 returned error can't find the container with id 390975a18f15f0846d490b7b711bd0a9298fe67d62626e789eecb38dbf7f103e
Jul 25 01:12:31.347463 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.347436    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:12:31.347463 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.347453    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:12:31.347594 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.347538    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:12:31.349777 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.349758    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 25 01:12:31.349861 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.349784    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2rr8l\""
Jul 25 01:12:31.350004 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.349990    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 25 01:12:31.350004 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.349997    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 25 01:12:31.350155 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.350089    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2flvd\""
Jul 25 01:12:31.350203 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.350188    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 25 01:12:31.535883 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.535845    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:31.536052 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:31.536003    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 25 01:12:31.536092 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:31.536068    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls podName:183e983e-4d28-47d2-be8e-36cfd1293dae nodeName:}" failed. No retries permitted until 2026-07-25 01:12:32.536054329 +0000 UTC m=+36.645863591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls") pod "dns-default-2bkbl" (UID: "183e983e-4d28-47d2-be8e-36cfd1293dae") : secret "dns-default-metrics-tls" not found
Jul 25 01:12:31.575007 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.574929    2572 generic.go:356] "Generic (PLEG): container finished" podID="036736e7-e135-4c08-be7a-8ac7d1975cdf" containerID="d1131d8ee94c2ba167250e9336c2217c3202416f8828d4b579f477ea87274947" exitCode=0
Jul 25 01:12:31.575138 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.575005    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6nrz" event={"ID":"036736e7-e135-4c08-be7a-8ac7d1975cdf","Type":"ContainerDied","Data":"d1131d8ee94c2ba167250e9336c2217c3202416f8828d4b579f477ea87274947"}
Jul 25 01:12:31.576174 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.576155    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x62hp" event={"ID":"cc0d00f8-2138-4cf2-98bf-d1898e41ddd1","Type":"ContainerStarted","Data":"390975a18f15f0846d490b7b711bd0a9298fe67d62626e789eecb38dbf7f103e"}
Jul 25 01:12:31.636898 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:31.636865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:31.637043 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:31.637012    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 25 01:12:31.637043 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:31.637029    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f58d88f64-52gnj: secret "image-registry-tls" not found
Jul 25 01:12:31.637131 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:31.637093    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls podName:edddd9d4-f442-4543-ae8b-f73319f9fbd4 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:32.637074458 +0000 UTC m=+36.746883743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls") pod "image-registry-6f58d88f64-52gnj" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4") : secret "image-registry-tls" not found
Jul 25 01:12:32.547180 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:32.546995    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:32.547849 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:32.547148    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 25 01:12:32.547849 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:32.547296    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls podName:183e983e-4d28-47d2-be8e-36cfd1293dae nodeName:}" failed. No retries permitted until 2026-07-25 01:12:34.54727936 +0000 UTC m=+38.657088623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls") pod "dns-default-2bkbl" (UID: "183e983e-4d28-47d2-be8e-36cfd1293dae") : secret "dns-default-metrics-tls" not found
Jul 25 01:12:32.581433 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:32.581405    2572 generic.go:356] "Generic (PLEG): container finished" podID="036736e7-e135-4c08-be7a-8ac7d1975cdf" containerID="0eb40b534cf0eef47a24d80b84a7687832d796d933245dbdcd36af00bc6d7ace" exitCode=0
Jul 25 01:12:32.581587 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:32.581459    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6nrz" event={"ID":"036736e7-e135-4c08-be7a-8ac7d1975cdf","Type":"ContainerDied","Data":"0eb40b534cf0eef47a24d80b84a7687832d796d933245dbdcd36af00bc6d7ace"}
Jul 25 01:12:32.648546 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:32.648486    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:32.648745 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:32.648726    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 25 01:12:32.648821 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:32.648750    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f58d88f64-52gnj: secret "image-registry-tls" not found
Jul 25 01:12:32.648821 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:32.648809    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls podName:edddd9d4-f442-4543-ae8b-f73319f9fbd4 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:34.648791174 +0000 UTC m=+38.758600447 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls") pod "image-registry-6f58d88f64-52gnj" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4") : secret "image-registry-tls" not found
Jul 25 01:12:33.588037 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:33.587994    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k6nrz" event={"ID":"036736e7-e135-4c08-be7a-8ac7d1975cdf","Type":"ContainerStarted","Data":"f39f3df58c2ab7d2f8511a8595228169ab6a00cae74d72cff95f2045e3932912"}
Jul 25 01:12:33.611174 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:33.611018    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k6nrz" podStartSLOduration=7.322676097 podStartE2EDuration="37.610998156s" totalImagesPullingTime="30.288322059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:33.609813891 +0000 UTC m=+37.719623179" watchObservedRunningTime="2026-07-25 01:12:33.610998156 +0000 UTC m=+37.720807443"
Jul 25 01:12:34.565406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:34.565369    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:34.565600 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:34.565503    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 25 01:12:34.565600 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:34.565574    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls podName:183e983e-4d28-47d2-be8e-36cfd1293dae nodeName:}" failed. No retries permitted until 2026-07-25 01:12:38.565553731 +0000 UTC m=+42.675363014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls") pod "dns-default-2bkbl" (UID: "183e983e-4d28-47d2-be8e-36cfd1293dae") : secret "dns-default-metrics-tls" not found
Jul 25 01:12:34.591318 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:34.591289    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x62hp" event={"ID":"cc0d00f8-2138-4cf2-98bf-d1898e41ddd1","Type":"ContainerStarted","Data":"26ccdd075a27f4daf2e24767e037d428cd5cc77c1fa50d55bfc0c69d96fcffbf"}
Jul 25 01:12:34.665706 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:34.665675    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:34.665841 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:34.665824    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 25 01:12:34.665841 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:34.665838    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f58d88f64-52gnj: secret "image-registry-tls" not found
Jul 25 01:12:34.665912 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:34.665892    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls podName:edddd9d4-f442-4543-ae8b-f73319f9fbd4 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:38.66587501 +0000 UTC m=+42.775684294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls") pod "image-registry-6f58d88f64-52gnj" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4") : secret "image-registry-tls" not found
Jul 25 01:12:38.589841 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:38.589803    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:38.590215 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:38.589954    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 25 01:12:38.590215 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:38.590016    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls podName:183e983e-4d28-47d2-be8e-36cfd1293dae nodeName:}" failed. No retries permitted until 2026-07-25 01:12:46.590000217 +0000 UTC m=+50.699809480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls") pod "dns-default-2bkbl" (UID: "183e983e-4d28-47d2-be8e-36cfd1293dae") : secret "dns-default-metrics-tls" not found
Jul 25 01:12:38.690437 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:38.690411    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:38.690571 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:38.690530    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 25 01:12:38.690571 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:38.690542    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f58d88f64-52gnj: secret "image-registry-tls" not found
Jul 25 01:12:38.690657 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:38.690589    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls podName:edddd9d4-f442-4543-ae8b-f73319f9fbd4 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:46.690574588 +0000 UTC m=+50.800383849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls") pod "image-registry-6f58d88f64-52gnj" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4") : secret "image-registry-tls" not found
Jul 25 01:12:41.203812 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.203759    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x62hp" podStartSLOduration=8.612154737000001 podStartE2EDuration="11.203743178s" totalImagesPullingTime="2.591588441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:34.60627873 +0000 UTC m=+38.716088013" watchObservedRunningTime="2026-07-25 01:12:41.203743178 +0000 UTC m=+45.313552461"
Jul 25 01:12:41.204290 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.204238    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-js2ft"]
Jul 25 01:12:41.237944 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.237919    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-js2ft"]
Jul 25 01:12:41.238081 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.238016    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.240525 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.240506    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 25 01:12:41.240653 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.240512    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 25 01:12:41.240653 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.240545    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mmjlg\""
Jul 25 01:12:41.241451 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.241434    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 25 01:12:41.241544 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.241435    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 25 01:12:41.409145 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.409117    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dff2d71-7a7b-4918-b55c-614b1a8d2065-data-volume\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.409145 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.409143    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bd58\" (UniqueName: \"kubernetes.io/projected/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-api-access-5bd58\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.409376 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.409180    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.409376 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.409271    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dff2d71-7a7b-4918-b55c-614b1a8d2065-crio-socket\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.409376 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.409307    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.510558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.510479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dff2d71-7a7b-4918-b55c-614b1a8d2065-data-volume\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.510558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.510513    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bd58\" (UniqueName: \"kubernetes.io/projected/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-api-access-5bd58\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.510558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.510551    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.510790 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.510582    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dff2d71-7a7b-4918-b55c-614b1a8d2065-crio-socket\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.510790 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.510599    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.510790 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:41.510686    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 25 01:12:41.510790 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:41.510739    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls podName:4dff2d71-7a7b-4918-b55c-614b1a8d2065 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:42.010726441 +0000 UTC m=+46.120535703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls") pod "insights-runtime-extractor-js2ft" (UID: "4dff2d71-7a7b-4918-b55c-614b1a8d2065") : secret "insights-runtime-extractor-tls" not found
Jul 25 01:12:41.510790 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:41.510684    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 25 01:12:41.510790 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:41.510782    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm podName:4dff2d71-7a7b-4918-b55c-614b1a8d2065 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:42.010772805 +0000 UTC m=+46.120582067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm") pod "insights-runtime-extractor-js2ft" (UID: "4dff2d71-7a7b-4918-b55c-614b1a8d2065") : configmap "kube-rbac-proxy" not found
Jul 25 01:12:41.511043 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.510800    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4dff2d71-7a7b-4918-b55c-614b1a8d2065-crio-socket\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.511043 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.510805    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4dff2d71-7a7b-4918-b55c-614b1a8d2065-data-volume\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:41.520561 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:41.520532    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bd58\" (UniqueName: \"kubernetes.io/projected/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-api-access-5bd58\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:42.014335 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:42.014289    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:42.014503 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:42.014365    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:42.014503 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:42.014422    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 25 01:12:42.014503 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:42.014502    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm podName:4dff2d71-7a7b-4918-b55c-614b1a8d2065 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:43.014484637 +0000 UTC m=+47.124293913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm") pod "insights-runtime-extractor-js2ft" (UID: "4dff2d71-7a7b-4918-b55c-614b1a8d2065") : configmap "kube-rbac-proxy" not found
Jul 25 01:12:42.014634 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:42.014510    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 25 01:12:42.014634 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:42.014567    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls podName:4dff2d71-7a7b-4918-b55c-614b1a8d2065 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:43.014554918 +0000 UTC m=+47.124364179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls") pod "insights-runtime-extractor-js2ft" (UID: "4dff2d71-7a7b-4918-b55c-614b1a8d2065") : secret "insights-runtime-extractor-tls" not found
Jul 25 01:12:42.898673 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:42.898642    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-62bbc"]
Jul 25 01:12:42.931768 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:42.931741    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-62bbc"]
Jul 25 01:12:42.931912 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:42.931827    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-62bbc"
Jul 25 01:12:42.934694 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:42.934675    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 25 01:12:42.935611 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:42.935592    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ps95h\""
Jul 25 01:12:42.935667 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:42.935630    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 25 01:12:42.935916 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:42.935899    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 25 01:12:42.935994 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:42.935972    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 25 01:12:43.022579 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.022549    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:43.022716 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.022598    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:43.022716 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:43.022669    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 25 01:12:43.022716 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:43.022710    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 25 01:12:43.022909 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:43.022733    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm podName:4dff2d71-7a7b-4918-b55c-614b1a8d2065 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:45.022716753 +0000 UTC m=+49.132526015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm") pod "insights-runtime-extractor-js2ft" (UID: "4dff2d71-7a7b-4918-b55c-614b1a8d2065") : configmap "kube-rbac-proxy" not found
Jul 25 01:12:43.022909 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:43.022754    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls podName:4dff2d71-7a7b-4918-b55c-614b1a8d2065 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:45.022746002 +0000 UTC m=+49.132555264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls") pod "insights-runtime-extractor-js2ft" (UID: "4dff2d71-7a7b-4918-b55c-614b1a8d2065") : secret "insights-runtime-extractor-tls" not found
Jul 25 01:12:43.123240 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.123213    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fe375059-79d3-419f-a2eb-95a4f68589b8-signing-key\") pod \"service-ca-658d7b7c78-62bbc\" (UID: \"fe375059-79d3-419f-a2eb-95a4f68589b8\") " pod="openshift-service-ca/service-ca-658d7b7c78-62bbc"
Jul 25 01:12:43.123435 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.123266    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fe375059-79d3-419f-a2eb-95a4f68589b8-signing-cabundle\") pod \"service-ca-658d7b7c78-62bbc\" (UID: \"fe375059-79d3-419f-a2eb-95a4f68589b8\") " pod="openshift-service-ca/service-ca-658d7b7c78-62bbc"
Jul 25 01:12:43.123435 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.123286    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsmkf\" (UniqueName: \"kubernetes.io/projected/fe375059-79d3-419f-a2eb-95a4f68589b8-kube-api-access-wsmkf\") pod \"service-ca-658d7b7c78-62bbc\" (UID: \"fe375059-79d3-419f-a2eb-95a4f68589b8\") " pod="openshift-service-ca/service-ca-658d7b7c78-62bbc"
Jul 25 01:12:43.224200 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.224130    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fe375059-79d3-419f-a2eb-95a4f68589b8-signing-key\") pod \"service-ca-658d7b7c78-62bbc\" (UID: \"fe375059-79d3-419f-a2eb-95a4f68589b8\") " pod="openshift-service-ca/service-ca-658d7b7c78-62bbc"
Jul 25 01:12:43.224200 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.224193    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fe375059-79d3-419f-a2eb-95a4f68589b8-signing-cabundle\") pod \"service-ca-658d7b7c78-62bbc\" (UID: \"fe375059-79d3-419f-a2eb-95a4f68589b8\") " pod="openshift-service-ca/service-ca-658d7b7c78-62bbc"
Jul 25 01:12:43.224395 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.224216    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsmkf\" (UniqueName: \"kubernetes.io/projected/fe375059-79d3-419f-a2eb-95a4f68589b8-kube-api-access-wsmkf\") pod \"service-ca-658d7b7c78-62bbc\" (UID: \"fe375059-79d3-419f-a2eb-95a4f68589b8\") " pod="openshift-service-ca/service-ca-658d7b7c78-62bbc"
Jul 25 01:12:43.224903 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.224874    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fe375059-79d3-419f-a2eb-95a4f68589b8-signing-cabundle\") pod \"service-ca-658d7b7c78-62bbc\" (UID: \"fe375059-79d3-419f-a2eb-95a4f68589b8\") " pod="openshift-service-ca/service-ca-658d7b7c78-62bbc"
Jul 25 01:12:43.226422 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.226400    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fe375059-79d3-419f-a2eb-95a4f68589b8-signing-key\") pod \"service-ca-658d7b7c78-62bbc\" (UID: \"fe375059-79d3-419f-a2eb-95a4f68589b8\") " pod="openshift-service-ca/service-ca-658d7b7c78-62bbc"
Jul 25 01:12:43.232060 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.232034    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsmkf\" (UniqueName: \"kubernetes.io/projected/fe375059-79d3-419f-a2eb-95a4f68589b8-kube-api-access-wsmkf\") pod \"service-ca-658d7b7c78-62bbc\" (UID: \"fe375059-79d3-419f-a2eb-95a4f68589b8\") " pod="openshift-service-ca/service-ca-658d7b7c78-62bbc"
Jul 25 01:12:43.240815 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.240797    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-62bbc"
Jul 25 01:12:43.377876 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.377819    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-62bbc"]
Jul 25 01:12:43.381907 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:12:43.381882    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe375059_79d3_419f_a2eb_95a4f68589b8.slice/crio-938c028b33d7e84f1b5a2fe467f1aa1866ead0f39b322cd34068f57692e08200 WatchSource:0}: Error finding container 938c028b33d7e84f1b5a2fe467f1aa1866ead0f39b322cd34068f57692e08200: Status 404 returned error can't find the container with id 938c028b33d7e84f1b5a2fe467f1aa1866ead0f39b322cd34068f57692e08200
Jul 25 01:12:43.612399 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:43.612310    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-62bbc" event={"ID":"fe375059-79d3-419f-a2eb-95a4f68589b8","Type":"ContainerStarted","Data":"938c028b33d7e84f1b5a2fe467f1aa1866ead0f39b322cd34068f57692e08200"}
Jul 25 01:12:44.000796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:44.000759    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fwsm2_0889cc33-fc59-4e79-a109-6e98bf53cc58/dns-node-resolver/0.log"
Jul 25 01:12:45.000736 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:45.000704    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jd9wg_373f9d31-a96b-4a50-b460-0716a102570f/node-ca/0.log"
Jul 25 01:12:45.038612 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:45.038567    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:45.039040 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:45.038648    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:45.039040 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:45.038684    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 25 01:12:45.039040 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:45.038742    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm podName:4dff2d71-7a7b-4918-b55c-614b1a8d2065 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:49.038727705 +0000 UTC m=+53.148536967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm") pod "insights-runtime-extractor-js2ft" (UID: "4dff2d71-7a7b-4918-b55c-614b1a8d2065") : configmap "kube-rbac-proxy" not found
Jul 25 01:12:45.039040 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:45.038786    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 25 01:12:45.039040 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:45.038843    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls podName:4dff2d71-7a7b-4918-b55c-614b1a8d2065 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:49.038828688 +0000 UTC m=+53.148637958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls") pod "insights-runtime-extractor-js2ft" (UID: "4dff2d71-7a7b-4918-b55c-614b1a8d2065") : secret "insights-runtime-extractor-tls" not found
Jul 25 01:12:46.000495 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:46.000462    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x62hp_cc0d00f8-2138-4cf2-98bf-d1898e41ddd1/serve-healthcheck-canary/0.log"
Jul 25 01:12:46.620073 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:46.620039    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-62bbc" event={"ID":"fe375059-79d3-419f-a2eb-95a4f68589b8","Type":"ContainerStarted","Data":"b6c8f791ae630f82c9055f61f5caa87306a1371a89a8d4eb38f1479d316810ac"}
Jul 25 01:12:46.643816 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:46.643765    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-62bbc" podStartSLOduration=2.422426522 podStartE2EDuration="4.6437514s" totalImagesPullingTime="2.221324878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:12:46.642983896 +0000 UTC m=+50.752793181" watchObservedRunningTime="2026-07-25 01:12:46.6437514 +0000 UTC m=+50.753560685"
Jul 25 01:12:46.652579 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:46.652552    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:12:46.652704 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:46.652683    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 25 01:12:46.652763 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:46.652755    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls podName:183e983e-4d28-47d2-be8e-36cfd1293dae nodeName:}" failed. No retries permitted until 2026-07-25 01:13:02.652737039 +0000 UTC m=+66.762546305 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls") pod "dns-default-2bkbl" (UID: "183e983e-4d28-47d2-be8e-36cfd1293dae") : secret "dns-default-metrics-tls" not found
Jul 25 01:12:46.753865 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:46.753836    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:12:46.753983 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:46.753964    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 25 01:12:46.753983 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:46.753983    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f58d88f64-52gnj: secret "image-registry-tls" not found
Jul 25 01:12:46.754098 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:46.754032    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls podName:edddd9d4-f442-4543-ae8b-f73319f9fbd4 nodeName:}" failed. No retries permitted until 2026-07-25 01:13:02.754017388 +0000 UTC m=+66.863826651 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls") pod "image-registry-6f58d88f64-52gnj" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4") : secret "image-registry-tls" not found
Jul 25 01:12:49.074149 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:49.074108    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:49.074656 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:49.074164    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:49.074656 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:49.074284    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 25 01:12:49.074656 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:12:49.074354    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls podName:4dff2d71-7a7b-4918-b55c-614b1a8d2065 nodeName:}" failed. No retries permitted until 2026-07-25 01:12:57.074339637 +0000 UTC m=+61.184148911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls") pod "insights-runtime-extractor-js2ft" (UID: "4dff2d71-7a7b-4918-b55c-614b1a8d2065") : secret "insights-runtime-extractor-tls" not found
Jul 25 01:12:49.074834 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:49.074797    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4dff2d71-7a7b-4918-b55c-614b1a8d2065-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:56.576422 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:56.576389    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vwkth"
Jul 25 01:12:57.128768 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:57.128734    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:57.130965 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:57.130938    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4dff2d71-7a7b-4918-b55c-614b1a8d2065-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-js2ft\" (UID: \"4dff2d71-7a7b-4918-b55c-614b1a8d2065\") " pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:57.149425 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:57.149399    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mmjlg\""
Jul 25 01:12:57.157349 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:57.157318    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-js2ft"
Jul 25 01:12:57.303541 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:57.303513    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-js2ft"]
Jul 25 01:12:57.306559 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:12:57.306531    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dff2d71_7a7b_4918_b55c_614b1a8d2065.slice/crio-3d7f901456d49c1ed28812842c13feb63b24f184a0550c198503a234abf7da73 WatchSource:0}: Error finding container 3d7f901456d49c1ed28812842c13feb63b24f184a0550c198503a234abf7da73: Status 404 returned error can't find the container with id 3d7f901456d49c1ed28812842c13feb63b24f184a0550c198503a234abf7da73
Jul 25 01:12:57.647342 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:12:57.647294    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-js2ft" event={"ID":"4dff2d71-7a7b-4918-b55c-614b1a8d2065","Type":"ContainerStarted","Data":"3d7f901456d49c1ed28812842c13feb63b24f184a0550c198503a234abf7da73"}
Jul 25 01:13:01.059373 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.059315    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:13:01.061930 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.061906    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 25 01:13:01.072309 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.072269    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0a939b-732b-4313-b0f9-2ce5feb9cc5c-metrics-certs\") pod \"network-metrics-daemon-nmt2v\" (UID: \"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c\") " pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:13:01.160451 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.160409    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:13:01.162998 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.162970    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 25 01:13:01.173421 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.173387    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6c8fc4b5-4d66-4ed8-948b-a9a57931649e-original-pull-secret\") pod \"global-pull-secret-syncer-hkv52\" (UID: \"6c8fc4b5-4d66-4ed8-948b-a9a57931649e\") " pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:13:01.261693 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.261655    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrfkm\" (UniqueName: \"kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm\") pod \"network-check-target-qqdd4\" (UID: \"df43e33a-d9bd-4d2e-9a95-f7487dd3c442\") " pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:13:01.264275 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.264257    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 25 01:13:01.274399 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.274377    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 25 01:13:01.285121 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.285094    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrfkm\" (UniqueName: \"kubernetes.io/projected/df43e33a-d9bd-4d2e-9a95-f7487dd3c442-kube-api-access-jrfkm\") pod \"network-check-target-qqdd4\" (UID: \"df43e33a-d9bd-4d2e-9a95-f7487dd3c442\") " pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:13:01.359747 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.359682    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2rr8l\""
Jul 25 01:13:01.364828 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.364807    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2flvd\""
Jul 25 01:13:01.367624 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.367597    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:13:01.367741 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.367627    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hkv52"
Jul 25 01:13:01.372728 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:01.372705    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nmt2v"
Jul 25 01:13:02.333202 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.333175    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hkv52"]
Jul 25 01:13:02.337077 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:02.337049    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8fc4b5_4d66_4ed8_948b_a9a57931649e.slice/crio-02d9db24dbbce7812ca0d19688944d3a3c6f2b54ff4f9d5549a8cab327bf2527 WatchSource:0}: Error finding container 02d9db24dbbce7812ca0d19688944d3a3c6f2b54ff4f9d5549a8cab327bf2527: Status 404 returned error can't find the container with id 02d9db24dbbce7812ca0d19688944d3a3c6f2b54ff4f9d5549a8cab327bf2527
Jul 25 01:13:02.559078 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.559000    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qqdd4"]
Jul 25 01:13:02.561741 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.561713    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nmt2v"]
Jul 25 01:13:02.562629 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:02.562587    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf43e33a_d9bd_4d2e_9a95_f7487dd3c442.slice/crio-b2e1bff041cee55ca922ff79caa5a63925de47847bde6d1b9caa5ff9df32819d WatchSource:0}: Error finding container b2e1bff041cee55ca922ff79caa5a63925de47847bde6d1b9caa5ff9df32819d: Status 404 returned error can't find the container with id b2e1bff041cee55ca922ff79caa5a63925de47847bde6d1b9caa5ff9df32819d
Jul 25 01:13:02.565859 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:02.565835    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0a939b_732b_4313_b0f9_2ce5feb9cc5c.slice/crio-4e450775df82a35715c5fc3a72eb0c9679d1564acdaf055e3d466b9816053106 WatchSource:0}: Error finding container 4e450775df82a35715c5fc3a72eb0c9679d1564acdaf055e3d466b9816053106: Status 404 returned error can't find the container with id 4e450775df82a35715c5fc3a72eb0c9679d1564acdaf055e3d466b9816053106
Jul 25 01:13:02.661593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.661561    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qqdd4" event={"ID":"df43e33a-d9bd-4d2e-9a95-f7487dd3c442","Type":"ContainerStarted","Data":"b2e1bff041cee55ca922ff79caa5a63925de47847bde6d1b9caa5ff9df32819d"}
Jul 25 01:13:02.662659 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.662635    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hkv52" event={"ID":"6c8fc4b5-4d66-4ed8-948b-a9a57931649e","Type":"ContainerStarted","Data":"02d9db24dbbce7812ca0d19688944d3a3c6f2b54ff4f9d5549a8cab327bf2527"}
Jul 25 01:13:02.663892 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.663869    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-js2ft" event={"ID":"4dff2d71-7a7b-4918-b55c-614b1a8d2065","Type":"ContainerStarted","Data":"028df1178f8783ce06f060bf1d9ba9c3c41b8be1b769c70ef251e16dc4315ad6"}
Jul 25 01:13:02.664878 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.664859    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmt2v" event={"ID":"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c","Type":"ContainerStarted","Data":"4e450775df82a35715c5fc3a72eb0c9679d1564acdaf055e3d466b9816053106"}
Jul 25 01:13:02.674192 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.674174    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:13:02.676532 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.676511    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/183e983e-4d28-47d2-be8e-36cfd1293dae-metrics-tls\") pod \"dns-default-2bkbl\" (UID: \"183e983e-4d28-47d2-be8e-36cfd1293dae\") " pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:13:02.775278 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.775235    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:13:02.778315 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.778282    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls\") pod \"image-registry-6f58d88f64-52gnj\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") " pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:13:02.951659 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.951626    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fxvm2\""
Jul 25 01:13:02.957864 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.957836    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:13:02.985916 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.985886    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g77zs\""
Jul 25 01:13:02.993952 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:02.993925    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:13:03.148387 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:03.148310    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2bkbl"]
Jul 25 01:13:03.155643 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:03.154461    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183e983e_4d28_47d2_be8e_36cfd1293dae.slice/crio-1c3f04ee5c5080f36aaa50c8aa7d1fb95e4e198e02a638e654588b699fb0817f WatchSource:0}: Error finding container 1c3f04ee5c5080f36aaa50c8aa7d1fb95e4e198e02a638e654588b699fb0817f: Status 404 returned error can't find the container with id 1c3f04ee5c5080f36aaa50c8aa7d1fb95e4e198e02a638e654588b699fb0817f
Jul 25 01:13:03.181055 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:03.180982    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f58d88f64-52gnj"]
Jul 25 01:13:03.185690 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:03.185651    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedddd9d4_f442_4543_ae8b_f73319f9fbd4.slice/crio-2f42c1170809cf019b21debb011a0647462c6b1b414170d0c04985fe91b72cda WatchSource:0}: Error finding container 2f42c1170809cf019b21debb011a0647462c6b1b414170d0c04985fe91b72cda: Status 404 returned error can't find the container with id 2f42c1170809cf019b21debb011a0647462c6b1b414170d0c04985fe91b72cda
Jul 25 01:13:03.671840 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:03.671797    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj" event={"ID":"edddd9d4-f442-4543-ae8b-f73319f9fbd4","Type":"ContainerStarted","Data":"a47df6247ad77a331be3d2e679a55e93682762d7e6fe80331cb7fccb0a31ede6"}
Jul 25 01:13:03.673011 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:03.671872    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj" event={"ID":"edddd9d4-f442-4543-ae8b-f73319f9fbd4","Type":"ContainerStarted","Data":"2f42c1170809cf019b21debb011a0647462c6b1b414170d0c04985fe91b72cda"}
Jul 25 01:13:03.673011 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:03.672723    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:13:03.678487 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:03.678461    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bkbl" event={"ID":"183e983e-4d28-47d2-be8e-36cfd1293dae","Type":"ContainerStarted","Data":"1c3f04ee5c5080f36aaa50c8aa7d1fb95e4e198e02a638e654588b699fb0817f"}
Jul 25 01:13:03.694658 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:03.694597    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj" podStartSLOduration=49.694577525 podStartE2EDuration="49.694577525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:03.693877655 +0000 UTC m=+67.803686944" watchObservedRunningTime="2026-07-25 01:13:03.694577525 +0000 UTC m=+67.804386812"
Jul 25 01:13:05.688481 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:05.688428    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmt2v" event={"ID":"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c","Type":"ContainerStarted","Data":"05521310df6cc79811e73aa797de0d8b95ae986fcb50d4b4655c3b336458968e"}
Jul 25 01:13:06.036178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.036078    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"]
Jul 25 01:13:06.063333 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.063284    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"]
Jul 25 01:13:06.063472 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.063421    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:06.066990 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.066965    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 25 01:13:06.067127 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.066968    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 25 01:13:06.067127 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.066968    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 25 01:13:06.067277 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.067201    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 25 01:13:06.070638 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.070620    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f58d88f64-52gnj"]
Jul 25 01:13:06.114182 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.114153    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n6sb\" (UniqueName: \"kubernetes.io/projected/42dcc803-bb38-47b7-bae8-5f92dbea4566-kube-api-access-2n6sb\") pod \"klusterlet-addon-workmgr-794f79c4f5-k8xww\" (UID: \"42dcc803-bb38-47b7-bae8-5f92dbea4566\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:06.114311 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.114188    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42dcc803-bb38-47b7-bae8-5f92dbea4566-tmp\") pod \"klusterlet-addon-workmgr-794f79c4f5-k8xww\" (UID: \"42dcc803-bb38-47b7-bae8-5f92dbea4566\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:06.114311 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.114212    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/42dcc803-bb38-47b7-bae8-5f92dbea4566-klusterlet-config\") pod \"klusterlet-addon-workmgr-794f79c4f5-k8xww\" (UID: \"42dcc803-bb38-47b7-bae8-5f92dbea4566\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:06.148253 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.148226    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-559c56d4b8-fnljn"]
Jul 25 01:13:06.168737 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.168709    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-559c56d4b8-fnljn"]
Jul 25 01:13:06.168864 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.168811    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.214876 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.214831    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42dcc803-bb38-47b7-bae8-5f92dbea4566-tmp\") pod \"klusterlet-addon-workmgr-794f79c4f5-k8xww\" (UID: \"42dcc803-bb38-47b7-bae8-5f92dbea4566\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:06.214876 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.214863    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/42dcc803-bb38-47b7-bae8-5f92dbea4566-klusterlet-config\") pod \"klusterlet-addon-workmgr-794f79c4f5-k8xww\" (UID: \"42dcc803-bb38-47b7-bae8-5f92dbea4566\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:06.215092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.214891    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-ca-trust-extracted\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.215092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.214918    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-image-registry-private-configuration\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.215092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.214943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-bound-sa-token\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.215092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.214970    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-trusted-ca\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.215092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.214999    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szw8v\" (UniqueName: \"kubernetes.io/projected/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-kube-api-access-szw8v\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.215386 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.215084    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-installation-pull-secrets\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.215386 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.215136    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n6sb\" (UniqueName: \"kubernetes.io/projected/42dcc803-bb38-47b7-bae8-5f92dbea4566-kube-api-access-2n6sb\") pod \"klusterlet-addon-workmgr-794f79c4f5-k8xww\" (UID: \"42dcc803-bb38-47b7-bae8-5f92dbea4566\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:06.215386 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.215154    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-registry-tls\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.215386 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.215169    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-registry-certificates\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.215386 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.215313    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42dcc803-bb38-47b7-bae8-5f92dbea4566-tmp\") pod \"klusterlet-addon-workmgr-794f79c4f5-k8xww\" (UID: \"42dcc803-bb38-47b7-bae8-5f92dbea4566\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:06.217208 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.217186    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/42dcc803-bb38-47b7-bae8-5f92dbea4566-klusterlet-config\") pod \"klusterlet-addon-workmgr-794f79c4f5-k8xww\" (UID: \"42dcc803-bb38-47b7-bae8-5f92dbea4566\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:06.224718 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.224694    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n6sb\" (UniqueName: \"kubernetes.io/projected/42dcc803-bb38-47b7-bae8-5f92dbea4566-kube-api-access-2n6sb\") pod \"klusterlet-addon-workmgr-794f79c4f5-k8xww\" (UID: \"42dcc803-bb38-47b7-bae8-5f92dbea4566\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:06.315513 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.315443    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-ca-trust-extracted\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.315513 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.315479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-image-registry-private-configuration\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.315513 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.315510    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-bound-sa-token\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.315745 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.315531    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-trusted-ca\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.315745 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.315551    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szw8v\" (UniqueName: \"kubernetes.io/projected/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-kube-api-access-szw8v\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.315859 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.315758    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-installation-pull-secrets\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.315859 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.315785    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-ca-trust-extracted\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.315859 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.315800    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-registry-tls\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.315859 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.315838    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-registry-certificates\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.316686 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.316663    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-trusted-ca\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.316799 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.316684    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-registry-certificates\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.318393 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.318369    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-registry-tls\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.318496 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.318372    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-image-registry-private-configuration\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.318552 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.318488    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-installation-pull-secrets\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.324599 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.324503    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-bound-sa-token\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.324922 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.324903    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szw8v\" (UniqueName: \"kubernetes.io/projected/4a2ab7fb-d629-472f-8fd6-5c6862caa9a3-kube-api-access-szw8v\") pod \"image-registry-559c56d4b8-fnljn\" (UID: \"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3\") " pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:06.373172 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.373145    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:06.477106 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:06.477079    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:09.874164 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:09.873911    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"]
Jul 25 01:13:09.879093 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:09.878716    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dcc803_bb38_47b7_bae8_5f92dbea4566.slice/crio-c303b2e7ff550933f833b655c68149bf06e95bdefe561e93c7bc0ee2a4959e0d WatchSource:0}: Error finding container c303b2e7ff550933f833b655c68149bf06e95bdefe561e93c7bc0ee2a4959e0d: Status 404 returned error can't find the container with id c303b2e7ff550933f833b655c68149bf06e95bdefe561e93c7bc0ee2a4959e0d
Jul 25 01:13:09.897613 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:09.897076    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-559c56d4b8-fnljn"]
Jul 25 01:13:09.901543 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:09.901507    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2ab7fb_d629_472f_8fd6_5c6862caa9a3.slice/crio-12c530bfcdc0bf9491da0575024ce0259b740a844026d9bb370ecbdf0487775a WatchSource:0}: Error finding container 12c530bfcdc0bf9491da0575024ce0259b740a844026d9bb370ecbdf0487775a: Status 404 returned error can't find the container with id 12c530bfcdc0bf9491da0575024ce0259b740a844026d9bb370ecbdf0487775a
Jul 25 01:13:10.706595 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.706519    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hkv52" event={"ID":"6c8fc4b5-4d66-4ed8-948b-a9a57931649e","Type":"ContainerStarted","Data":"6355fa3f0a5b9e60a6f132dbf070a938e45639cd47b16975f503617d6bd0c144"}
Jul 25 01:13:10.708496 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.708464    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bkbl" event={"ID":"183e983e-4d28-47d2-be8e-36cfd1293dae","Type":"ContainerStarted","Data":"0939e5b4fde81c07766d87447304de1ed9f40ce18157c51ec345d72f3ea3d6ff"}
Jul 25 01:13:10.708637 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.708513    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bkbl" event={"ID":"183e983e-4d28-47d2-be8e-36cfd1293dae","Type":"ContainerStarted","Data":"89a6789c2299a616659f6d3583dec969c2ac5995890160adde867fcff4279655"}
Jul 25 01:13:10.708637 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.708541    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:13:10.710232 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.710190    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-559c56d4b8-fnljn" event={"ID":"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3","Type":"ContainerStarted","Data":"29a251e0a6fa8cc848142aef334dd1d19834a142b7bb4df5e46c7acb292b0961"}
Jul 25 01:13:10.710385 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.710236    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-559c56d4b8-fnljn" event={"ID":"4a2ab7fb-d629-472f-8fd6-5c6862caa9a3","Type":"ContainerStarted","Data":"12c530bfcdc0bf9491da0575024ce0259b740a844026d9bb370ecbdf0487775a"}
Jul 25 01:13:10.710385 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.710263    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:10.712150 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.712122    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-js2ft" event={"ID":"4dff2d71-7a7b-4918-b55c-614b1a8d2065","Type":"ContainerStarted","Data":"647adb17143e3bf2501f6752ff9d7dcc9ff1df4670bfa59ac1b403d732373c40"}
Jul 25 01:13:10.714016 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.713985    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nmt2v" event={"ID":"1f0a939b-732b-4313-b0f9-2ce5feb9cc5c","Type":"ContainerStarted","Data":"15d1d2ac4e0d8000a78971b8824d50e0f380312ce84a9bc8b9a763b316165642"}
Jul 25 01:13:10.715291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.715272    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww" event={"ID":"42dcc803-bb38-47b7-bae8-5f92dbea4566","Type":"ContainerStarted","Data":"c303b2e7ff550933f833b655c68149bf06e95bdefe561e93c7bc0ee2a4959e0d"}
Jul 25 01:13:10.716736 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.716715    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qqdd4" event={"ID":"df43e33a-d9bd-4d2e-9a95-f7487dd3c442","Type":"ContainerStarted","Data":"fa5dd9a80a9ab3b4ef59b71dfff73c0d619a823c4fe635fe27b1ceef3c717174"}
Jul 25 01:13:10.716869 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.716849    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:13:10.737244 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.737194    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hkv52" podStartSLOduration=66.376534478 podStartE2EDuration="1m13.737174684s" totalImagesPullingTime="7.360640206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:10.720591777 +0000 UTC m=+74.830401062" watchObservedRunningTime="2026-07-25 01:13:10.737174684 +0000 UTC m=+74.846983968"
Jul 25 01:13:10.754825 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.754782    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2bkbl" podStartSLOduration=34.210231859 podStartE2EDuration="40.754770225s" totalImagesPullingTime="6.544538366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:10.736836117 +0000 UTC m=+74.846645403" watchObservedRunningTime="2026-07-25 01:13:10.754770225 +0000 UTC m=+74.864579509"
Jul 25 01:13:10.755255 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.755226    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-559c56d4b8-fnljn" podStartSLOduration=4.7552181959999995 podStartE2EDuration="4.755218196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:10.754539607 +0000 UTC m=+74.864348892" watchObservedRunningTime="2026-07-25 01:13:10.755218196 +0000 UTC m=+74.865027600"
Jul 25 01:13:10.773195 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.773141    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qqdd4" podStartSLOduration=66.638244351 podStartE2EDuration="1m13.773123161s" totalImagesPullingTime="7.13487881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:10.771379344 +0000 UTC m=+74.881188666" watchObservedRunningTime="2026-07-25 01:13:10.773123161 +0000 UTC m=+74.882932445"
Jul 25 01:13:10.787204 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:10.787156    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nmt2v" podStartSLOduration=72.777279751 podStartE2EDuration="1m14.787137647s" totalImagesPullingTime="2.009857896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:10.785817365 +0000 UTC m=+74.895626675" watchObservedRunningTime="2026-07-25 01:13:10.787137647 +0000 UTC m=+74.896946929"
Jul 25 01:13:12.236307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:12.236271    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs"]
Jul 25 01:13:12.246365 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:12.246104    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs"
Jul 25 01:13:12.250585 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:12.249214    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 25 01:13:12.250585 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:12.249285    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-q2vtn\""
Jul 25 01:13:12.251847 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:12.251791    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs"]
Jul 25 01:13:12.367207 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:12.367142    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3239601d-ad90-41f4-b504-375eb1a61156-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-hjcfs\" (UID: \"3239601d-ad90-41f4-b504-375eb1a61156\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs"
Jul 25 01:13:12.467778 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:12.467738    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3239601d-ad90-41f4-b504-375eb1a61156-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-hjcfs\" (UID: \"3239601d-ad90-41f4-b504-375eb1a61156\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs"
Jul 25 01:13:12.467961 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:13:12.467880    2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 25 01:13:12.468040 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:13:12.467964    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3239601d-ad90-41f4-b504-375eb1a61156-tls-certificates podName:3239601d-ad90-41f4-b504-375eb1a61156 nodeName:}" failed. No retries permitted until 2026-07-25 01:13:12.967939818 +0000 UTC m=+77.077749080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/3239601d-ad90-41f4-b504-375eb1a61156-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-hjcfs" (UID: "3239601d-ad90-41f4-b504-375eb1a61156") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 25 01:13:12.971733 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:12.971694    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3239601d-ad90-41f4-b504-375eb1a61156-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-hjcfs\" (UID: \"3239601d-ad90-41f4-b504-375eb1a61156\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs"
Jul 25 01:13:12.974612 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:12.974582    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3239601d-ad90-41f4-b504-375eb1a61156-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-hjcfs\" (UID: \"3239601d-ad90-41f4-b504-375eb1a61156\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs"
Jul 25 01:13:13.159967 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:13.159865    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs"
Jul 25 01:13:15.492007 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:15.491977    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs"]
Jul 25 01:13:15.495333 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:15.495289    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3239601d_ad90_41f4_b504_375eb1a61156.slice/crio-7ab4dacf01c6755458628b7a0af8318daa98c8da01b46cd4905e0b0d4d400ab7 WatchSource:0}: Error finding container 7ab4dacf01c6755458628b7a0af8318daa98c8da01b46cd4905e0b0d4d400ab7: Status 404 returned error can't find the container with id 7ab4dacf01c6755458628b7a0af8318daa98c8da01b46cd4905e0b0d4d400ab7
Jul 25 01:13:15.738590 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:15.738555    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-js2ft" event={"ID":"4dff2d71-7a7b-4918-b55c-614b1a8d2065","Type":"ContainerStarted","Data":"eff35b8e4d34d1e6e2ff5f6ccb46a5f72a7d1b18eee1c7e797e994fa4c6e6a99"}
Jul 25 01:13:15.739942 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:15.739916    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww" event={"ID":"42dcc803-bb38-47b7-bae8-5f92dbea4566","Type":"ContainerStarted","Data":"6a5d02c26338abb30fc4882320b4efd258f52f923adb079bc96c3edc3b28505e"}
Jul 25 01:13:15.740107 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:15.740093    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:15.741074 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:15.741041    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs" event={"ID":"3239601d-ad90-41f4-b504-375eb1a61156","Type":"ContainerStarted","Data":"7ab4dacf01c6755458628b7a0af8318daa98c8da01b46cd4905e0b0d4d400ab7"}
Jul 25 01:13:15.741986 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:15.741971    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww"
Jul 25 01:13:15.755817 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:15.755774    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-js2ft" podStartSLOduration=16.905688729 podStartE2EDuration="34.755760018s" totalImagesPullingTime="17.850071289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 01:12:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:15.755109155 +0000 UTC m=+79.864918458" watchObservedRunningTime="2026-07-25 01:13:15.755760018 +0000 UTC m=+79.865569614"
Jul 25 01:13:15.770048 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:15.770008    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-794f79c4f5-k8xww" podStartSLOduration=4.232780563 podStartE2EDuration="9.769996628s" totalImagesPullingTime="5.537216065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:13:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:15.769073089 +0000 UTC m=+79.878882375" watchObservedRunningTime="2026-07-25 01:13:15.769996628 +0000 UTC m=+79.879805912"
Jul 25 01:13:16.745227 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:16.745191    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs" event={"ID":"3239601d-ad90-41f4-b504-375eb1a61156","Type":"ContainerStarted","Data":"5a431b20a2c1ebc9166cf657bb9f74ca4ea7eca0392685d31824a39ad15b6615"}
Jul 25 01:13:16.745621 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:16.745545    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs"
Jul 25 01:13:16.749929 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:16.749905    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs"
Jul 25 01:13:16.761818 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:16.761778    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-hjcfs" podStartSLOduration=3.637550584 podStartE2EDuration="4.76176577s" totalImagesPullingTime="1.124215186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:16.760980045 +0000 UTC m=+80.870789330" watchObservedRunningTime="2026-07-25 01:13:16.76176577 +0000 UTC m=+80.871575054"
Jul 25 01:13:17.298348 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.298301    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"]
Jul 25 01:13:17.302717 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.302701    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.305261 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.305239    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 25 01:13:17.306081 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.306055    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 25 01:13:17.306081 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.306080    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 25 01:13:17.306246 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.306092    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 25 01:13:17.306297 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.306280    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xsmhm\""
Jul 25 01:13:17.306374 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.306359    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 25 01:13:17.314664 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.312471    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"]
Jul 25 01:13:17.383620 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.383587    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-654f5bcb7b-788cl"]
Jul 25 01:13:17.386605 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.386583    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.388835 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.388813    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 25 01:13:17.388835 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.388829    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 25 01:13:17.389002 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.388812    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 25 01:13:17.389002 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.388814    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 25 01:13:17.389126 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.389106    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 25 01:13:17.389180 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.389132    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8bx8f\""
Jul 25 01:13:17.389180 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.389166    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 25 01:13:17.389282 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.389164    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 25 01:13:17.395272 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.395251    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-654f5bcb7b-788cl"]
Jul 25 01:13:17.408158 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.408136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635ecd8f-a367-43ba-82bd-e81cf39131b7-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.408252 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.408169    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2z8v\" (UniqueName: \"kubernetes.io/projected/635ecd8f-a367-43ba-82bd-e81cf39131b7-kube-api-access-m2z8v\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.408252 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.408213    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/635ecd8f-a367-43ba-82bd-e81cf39131b7-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.408350 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.408249    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/635ecd8f-a367-43ba-82bd-e81cf39131b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.509452 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.509427    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-oauth-config\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.509579 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.509460    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/635ecd8f-a367-43ba-82bd-e81cf39131b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.509579 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.509484    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-oauth-serving-cert\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.509579 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.509528    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-serving-cert\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.509579 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.509576    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-service-ca\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.509724 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.509647    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgh8\" (UniqueName: \"kubernetes.io/projected/42b100c8-65dc-481b-9e38-81221ca55647-kube-api-access-szgh8\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.509724 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.509687    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635ecd8f-a367-43ba-82bd-e81cf39131b7-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.509724 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.509712    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-console-config\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.509849 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.509736    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2z8v\" (UniqueName: \"kubernetes.io/projected/635ecd8f-a367-43ba-82bd-e81cf39131b7-kube-api-access-m2z8v\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.509849 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.509788    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/635ecd8f-a367-43ba-82bd-e81cf39131b7-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.509991 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:13:17.509974    2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 25 01:13:17.510054 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:13:17.510045    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/635ecd8f-a367-43ba-82bd-e81cf39131b7-prometheus-operator-tls podName:635ecd8f-a367-43ba-82bd-e81cf39131b7 nodeName:}" failed. No retries permitted until 2026-07-25 01:13:18.010023482 +0000 UTC m=+82.119832747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/635ecd8f-a367-43ba-82bd-e81cf39131b7-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-rc8fx" (UID: "635ecd8f-a367-43ba-82bd-e81cf39131b7") : secret "prometheus-operator-tls" not found
Jul 25 01:13:17.510354 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.510318    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/635ecd8f-a367-43ba-82bd-e81cf39131b7-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.511822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.511799    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/635ecd8f-a367-43ba-82bd-e81cf39131b7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.520031 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.520001    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2z8v\" (UniqueName: \"kubernetes.io/projected/635ecd8f-a367-43ba-82bd-e81cf39131b7-kube-api-access-m2z8v\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:17.611141 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.611073    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-oauth-serving-cert\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.611141 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.611108    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-serving-cert\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.611141 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.611130    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-service-ca\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.611375 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.611167    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgh8\" (UniqueName: \"kubernetes.io/projected/42b100c8-65dc-481b-9e38-81221ca55647-kube-api-access-szgh8\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.611375 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.611194    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-console-config\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.611375 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.611262    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-oauth-config\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.611877 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.611855    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-service-ca\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.611970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.611858    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-oauth-serving-cert\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.611970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.611947    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-console-config\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.613569 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.613545    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-oauth-config\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.613655 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.613607    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-serving-cert\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.617998 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.617977    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgh8\" (UniqueName: \"kubernetes.io/projected/42b100c8-65dc-481b-9e38-81221ca55647-kube-api-access-szgh8\") pod \"console-654f5bcb7b-788cl\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") " pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.696017 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.695977    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:17.819450 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:17.819421    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-654f5bcb7b-788cl"]
Jul 25 01:13:17.822216 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:17.822188    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b100c8_65dc_481b_9e38_81221ca55647.slice/crio-b0f69d5d1dfa65489fc80880cb496e225c021fac5047572b2f537434cbd1f773 WatchSource:0}: Error finding container b0f69d5d1dfa65489fc80880cb496e225c021fac5047572b2f537434cbd1f773: Status 404 returned error can't find the container with id b0f69d5d1dfa65489fc80880cb496e225c021fac5047572b2f537434cbd1f773
Jul 25 01:13:18.014466 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:18.014441    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/635ecd8f-a367-43ba-82bd-e81cf39131b7-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:18.016740 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:18.016721    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/635ecd8f-a367-43ba-82bd-e81cf39131b7-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-rc8fx\" (UID: \"635ecd8f-a367-43ba-82bd-e81cf39131b7\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:18.217422 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:18.217300    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"
Jul 25 01:13:18.373616 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:18.373508    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-rc8fx"]
Jul 25 01:13:18.376620 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:18.376589    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635ecd8f_a367_43ba_82bd_e81cf39131b7.slice/crio-8e63c62ea0c4234aab7f337627aa76b9b7ca01420ce432b3df6da4e0866e994d WatchSource:0}: Error finding container 8e63c62ea0c4234aab7f337627aa76b9b7ca01420ce432b3df6da4e0866e994d: Status 404 returned error can't find the container with id 8e63c62ea0c4234aab7f337627aa76b9b7ca01420ce432b3df6da4e0866e994d
Jul 25 01:13:18.755389 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:18.755355    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx" event={"ID":"635ecd8f-a367-43ba-82bd-e81cf39131b7","Type":"ContainerStarted","Data":"8e63c62ea0c4234aab7f337627aa76b9b7ca01420ce432b3df6da4e0866e994d"}
Jul 25 01:13:18.756978 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:18.756921    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654f5bcb7b-788cl" event={"ID":"42b100c8-65dc-481b-9e38-81221ca55647","Type":"ContainerStarted","Data":"b0f69d5d1dfa65489fc80880cb496e225c021fac5047572b2f537434cbd1f773"}
Jul 25 01:13:20.723498 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:20.723468    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2bkbl"
Jul 25 01:13:21.770912 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:21.770857    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654f5bcb7b-788cl" event={"ID":"42b100c8-65dc-481b-9e38-81221ca55647","Type":"ContainerStarted","Data":"8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e"}
Jul 25 01:13:21.772580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:21.772550    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx" event={"ID":"635ecd8f-a367-43ba-82bd-e81cf39131b7","Type":"ContainerStarted","Data":"83d670e4b36d7a47766612ac51919c18c238df412c741a932889d0ea09967997"}
Jul 25 01:13:21.772697 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:21.772600    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx" event={"ID":"635ecd8f-a367-43ba-82bd-e81cf39131b7","Type":"ContainerStarted","Data":"a8b4c41cc0b4b5a975ab704367e94965236442b10da9d080e77201c99142fd18"}
Jul 25 01:13:21.787096 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:21.787049    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-654f5bcb7b-788cl" podStartSLOduration=1.8321556380000001 podStartE2EDuration="4.787032716s" totalImagesPullingTime="2.954877078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:21.786992067 +0000 UTC m=+85.896801352" watchObservedRunningTime="2026-07-25 01:13:21.787032716 +0000 UTC m=+85.896841993"
Jul 25 01:13:21.802713 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:21.802671    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-rc8fx" podStartSLOduration=2.41517157 podStartE2EDuration="4.802658554s" totalImagesPullingTime="2.387486984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:13:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:21.802154306 +0000 UTC m=+85.911963592" watchObservedRunningTime="2026-07-25 01:13:21.802658554 +0000 UTC m=+85.912467837"
Jul 25 01:13:23.656906 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.656873    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"]
Jul 25 01:13:23.660615 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.660591    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.664426 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.664405    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 25 01:13:23.664682 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.664659    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ztrfj\""
Jul 25 01:13:23.664851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.664665    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 25 01:13:23.675924 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.675857    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"]
Jul 25 01:13:23.685101 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.685079    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"]
Jul 25 01:13:23.688679 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.688658    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.691853 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.691832    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 25 01:13:23.691983 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.691897    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 25 01:13:23.692158 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.691909    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hvclx\""
Jul 25 01:13:23.692236 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.692057    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 25 01:13:23.703193 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.703165    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"]
Jul 25 01:13:23.707945 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.707920    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z2kt7"]
Jul 25 01:13:23.711673 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.711653    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.713963 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.713944    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 25 01:13:23.714292 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.714100    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 25 01:13:23.714445 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.713960    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 25 01:13:23.714605 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.714015    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z8cbs\""
Jul 25 01:13:23.760492 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.760468    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ff29538-913e-4789-bc77-ba85057fb62c-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.760636 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.760499    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b33f1931-861a-4b78-b7d3-5433ad272a0f-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.760636 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.760515    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29n8m\" (UniqueName: \"kubernetes.io/projected/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-api-access-29n8m\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.760636 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.760534    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b33f1931-861a-4b78-b7d3-5433ad272a0f-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.760636 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.760612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ff29538-913e-4789-bc77-ba85057fb62c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.760836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.760697    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ff29538-913e-4789-bc77-ba85057fb62c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.760836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.760730    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.760836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.760768    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqgbb\" (UniqueName: \"kubernetes.io/projected/5ff29538-913e-4789-bc77-ba85057fb62c-kube-api-access-dqgbb\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.760836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.760795    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.761030 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.760833    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.861841 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.861804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ff29538-913e-4789-bc77-ba85057fb62c-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.861841 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.861837    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d84a197-6a94-48f6-825e-16181f3b44d5-sys\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.862058 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.861857    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b33f1931-861a-4b78-b7d3-5433ad272a0f-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.862058 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.861989    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29n8m\" (UniqueName: \"kubernetes.io/projected/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-api-access-29n8m\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.862058 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862038    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b33f1931-861a-4b78-b7d3-5433ad272a0f-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.862214 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862084    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ff29538-913e-4789-bc77-ba85057fb62c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.862214 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.862214 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862154    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txdpz\" (UniqueName: \"kubernetes.io/projected/4d84a197-6a94-48f6-825e-16181f3b44d5-kube-api-access-txdpz\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.862214 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862189    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-tls\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.862491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862223    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ff29538-913e-4789-bc77-ba85057fb62c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.862491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862248    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b33f1931-861a-4b78-b7d3-5433ad272a0f-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.862491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862252    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d84a197-6a94-48f6-825e-16181f3b44d5-root\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.862491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862298    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.862491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d84a197-6a94-48f6-825e-16181f3b44d5-metrics-client-ca\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.862491 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:13:23.862429    2572 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 25 01:13:23.862775 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:13:23.862547    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff29538-913e-4789-bc77-ba85057fb62c-openshift-state-metrics-tls podName:5ff29538-913e-4789-bc77-ba85057fb62c nodeName:}" failed. No retries permitted until 2026-07-25 01:13:24.36248032 +0000 UTC m=+88.472289589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5ff29538-913e-4789-bc77-ba85057fb62c-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-6fqm5" (UID: "5ff29538-913e-4789-bc77-ba85057fb62c") : secret "openshift-state-metrics-tls" not found
Jul 25 01:13:23.862775 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862586    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.862775 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862618    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ff29538-913e-4789-bc77-ba85057fb62c-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.862929 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862621    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqgbb\" (UniqueName: \"kubernetes.io/projected/5ff29538-913e-4789-bc77-ba85057fb62c-kube-api-access-dqgbb\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.862929 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862901    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.863022 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862932    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-wtmp\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.863022 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.862980    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.863022 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.863006    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-textfile\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.863201 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.863120    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b33f1931-861a-4b78-b7d3-5433ad272a0f-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.863201 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:13:23.863148    2572 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 25 01:13:23.863296 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:13:23.863209    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-tls podName:b33f1931-861a-4b78-b7d3-5433ad272a0f nodeName:}" failed. No retries permitted until 2026-07-25 01:13:24.363193421 +0000 UTC m=+88.473002684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-77n8k" (UID: "b33f1931-861a-4b78-b7d3-5433ad272a0f") : secret "kube-state-metrics-tls" not found
Jul 25 01:13:23.863696 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.863673    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.864872 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.864851    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ff29538-913e-4789-bc77-ba85057fb62c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.865050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.865032    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.879270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.879244    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqgbb\" (UniqueName: \"kubernetes.io/projected/5ff29538-913e-4789-bc77-ba85057fb62c-kube-api-access-dqgbb\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:23.879505 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.879486    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29n8m\" (UniqueName: \"kubernetes.io/projected/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-api-access-29n8m\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:23.964318 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964252    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d84a197-6a94-48f6-825e-16181f3b44d5-root\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964318 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964282    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d84a197-6a94-48f6-825e-16181f3b44d5-metrics-client-ca\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964318 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964306    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964392    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d84a197-6a94-48f6-825e-16181f3b44d5-root\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964508    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-wtmp\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964660 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964557    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-textfile\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964660 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964615    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d84a197-6a94-48f6-825e-16181f3b44d5-sys\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964766 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964676    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-wtmp\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964766 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964766 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964733    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdpz\" (UniqueName: \"kubernetes.io/projected/4d84a197-6a94-48f6-825e-16181f3b44d5-kube-api-access-txdpz\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964963 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964774    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-tls\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964963 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964841    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d84a197-6a94-48f6-825e-16181f3b44d5-metrics-client-ca\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964963 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964873    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964963 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964901    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d84a197-6a94-48f6-825e-16181f3b44d5-sys\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.964963 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.964950    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-textfile\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.966912 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.966876    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.966996 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.966919    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d84a197-6a94-48f6-825e-16181f3b44d5-node-exporter-tls\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:23.973084 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:23.973066    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdpz\" (UniqueName: \"kubernetes.io/projected/4d84a197-6a94-48f6-825e-16181f3b44d5-kube-api-access-txdpz\") pod \"node-exporter-z2kt7\" (UID: \"4d84a197-6a94-48f6-825e-16181f3b44d5\") " pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:24.025613 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:24.025583    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z2kt7"
Jul 25 01:13:24.038826 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:24.038797    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d84a197_6a94_48f6_825e_16181f3b44d5.slice/crio-0187e08ffe8c802d3d09c1f9e45d0ebdc9030a6dfa029cbae42703d9be8b0436 WatchSource:0}: Error finding container 0187e08ffe8c802d3d09c1f9e45d0ebdc9030a6dfa029cbae42703d9be8b0436: Status 404 returned error can't find the container with id 0187e08ffe8c802d3d09c1f9e45d0ebdc9030a6dfa029cbae42703d9be8b0436
Jul 25 01:13:24.369018 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:24.368928    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ff29538-913e-4789-bc77-ba85057fb62c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:24.369018 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:24.368992    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:24.371234 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:24.371203    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ff29538-913e-4789-bc77-ba85057fb62c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-6fqm5\" (UID: \"5ff29538-913e-4789-bc77-ba85057fb62c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:24.371364 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:24.371263    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b33f1931-861a-4b78-b7d3-5433ad272a0f-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-77n8k\" (UID: \"b33f1931-861a-4b78-b7d3-5433ad272a0f\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:24.571198 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:24.571168    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"
Jul 25 01:13:24.602374 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:24.602348    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"
Jul 25 01:13:24.740842 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:24.739106    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5"]
Jul 25 01:13:24.785184 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:24.785152    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k"]
Jul 25 01:13:24.785883 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:24.785852    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z2kt7" event={"ID":"4d84a197-6a94-48f6-825e-16181f3b44d5","Type":"ContainerStarted","Data":"0187e08ffe8c802d3d09c1f9e45d0ebdc9030a6dfa029cbae42703d9be8b0436"}
Jul 25 01:13:24.924663 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:24.924625    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff29538_913e_4789_bc77_ba85057fb62c.slice/crio-cf286017c0016aacec9e25390872837a7818a736f135e43360ec9c4779bcb2e0 WatchSource:0}: Error finding container cf286017c0016aacec9e25390872837a7818a736f135e43360ec9c4779bcb2e0: Status 404 returned error can't find the container with id cf286017c0016aacec9e25390872837a7818a736f135e43360ec9c4779bcb2e0
Jul 25 01:13:24.925276 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:24.925236    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33f1931_861a_4b78_b7d3_5433ad272a0f.slice/crio-1fce470f8044f90edf51e83ed651cce672a981fc699e1c0c05dc005aa99dc091 WatchSource:0}: Error finding container 1fce470f8044f90edf51e83ed651cce672a981fc699e1c0c05dc005aa99dc091: Status 404 returned error can't find the container with id 1fce470f8044f90edf51e83ed651cce672a981fc699e1c0c05dc005aa99dc091
Jul 25 01:13:25.792966 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:25.792929    2572 generic.go:356] "Generic (PLEG): container finished" podID="4d84a197-6a94-48f6-825e-16181f3b44d5" containerID="9671edf8e7fea5a86a286fcdeba375657c49f6135e9d580cae660fb2654f009d" exitCode=0
Jul 25 01:13:25.793436 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:25.793020    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z2kt7" event={"ID":"4d84a197-6a94-48f6-825e-16181f3b44d5","Type":"ContainerDied","Data":"9671edf8e7fea5a86a286fcdeba375657c49f6135e9d580cae660fb2654f009d"}
Jul 25 01:13:25.795840 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:25.795714    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5" event={"ID":"5ff29538-913e-4789-bc77-ba85057fb62c","Type":"ContainerStarted","Data":"53a6a35cab71baba86b5df9b9166ab57ebd31d12f6909b3f571c6974a3b3990b"}
Jul 25 01:13:25.795840 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:25.795775    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5" event={"ID":"5ff29538-913e-4789-bc77-ba85057fb62c","Type":"ContainerStarted","Data":"2424139dfbb06e1241fd154ef0488a3a0e6ad31643a83abe6f1ef4afd17107aa"}
Jul 25 01:13:25.795840 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:25.795804    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5" event={"ID":"5ff29538-913e-4789-bc77-ba85057fb62c","Type":"ContainerStarted","Data":"cf286017c0016aacec9e25390872837a7818a736f135e43360ec9c4779bcb2e0"}
Jul 25 01:13:25.797160 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:25.797135    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k" event={"ID":"b33f1931-861a-4b78-b7d3-5433ad272a0f","Type":"ContainerStarted","Data":"1fce470f8044f90edf51e83ed651cce672a981fc699e1c0c05dc005aa99dc091"}
Jul 25 01:13:26.698931 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:26.698903    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:13:26.805048 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:26.804811    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z2kt7" event={"ID":"4d84a197-6a94-48f6-825e-16181f3b44d5","Type":"ContainerStarted","Data":"87228c457df0c911d2205b038239b504b1b404e51a7e2d503965e8b93a2ec757"}
Jul 25 01:13:26.805048 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:26.804877    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z2kt7" event={"ID":"4d84a197-6a94-48f6-825e-16181f3b44d5","Type":"ContainerStarted","Data":"af472a64eedc0b3bb135f1307196522e69d8b083a3615de78fe36975d49c4b32"}
Jul 25 01:13:26.807555 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:26.807527    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5" event={"ID":"5ff29538-913e-4789-bc77-ba85057fb62c","Type":"ContainerStarted","Data":"89be7e40d09ae211566455f5b5d492b234f1def25adafb86e08cbdd798b0036c"}
Jul 25 01:13:26.810025 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:26.809999    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k" event={"ID":"b33f1931-861a-4b78-b7d3-5433ad272a0f","Type":"ContainerStarted","Data":"340306e4a350216b7e862cc3ca1f7a203f2a5129bdbdd6b7f7bc7a957f1439f7"}
Jul 25 01:13:26.810200 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:26.810177    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k" event={"ID":"b33f1931-861a-4b78-b7d3-5433ad272a0f","Type":"ContainerStarted","Data":"95845e168b9203761deb2ba2f8f76b62b742b78073896730cc734ca72013e1c6"}
Jul 25 01:13:26.810289 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:26.810215    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k" event={"ID":"b33f1931-861a-4b78-b7d3-5433ad272a0f","Type":"ContainerStarted","Data":"0d962d868165aac7120fe205ca458e4e9392f2c6538453fc2a4bd0905c625b49"}
Jul 25 01:13:26.829414 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:26.829363    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z2kt7" podStartSLOduration=2.834329632 podStartE2EDuration="3.829350766s" totalImagesPullingTime="928.888977ms" totalInitContainerRuntime="66.132157ms" isStatefulPod=true podCreationTimestamp="2026-07-25 01:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:26.828955198 +0000 UTC m=+90.938764483" watchObservedRunningTime="2026-07-25 01:13:26.829350766 +0000 UTC m=+90.939160027"
Jul 25 01:13:26.856681 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:26.856633    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-6fqm5" podStartSLOduration=2.525180363 podStartE2EDuration="3.856620169s" totalImagesPullingTime="1.331439806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:26.856464988 +0000 UTC m=+90.966274271" watchObservedRunningTime="2026-07-25 01:13:26.856620169 +0000 UTC m=+90.966429456"
Jul 25 01:13:26.896319 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:26.896263    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-77n8k" podStartSLOduration=2.431269263 podStartE2EDuration="3.896248608s" totalImagesPullingTime="1.464979345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:26.89466458 +0000 UTC m=+91.004473855" watchObservedRunningTime="2026-07-25 01:13:26.896248608 +0000 UTC m=+91.006057891"
Jul 25 01:13:27.696606 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:27.696566    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:27.696779 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:27.696635    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:27.701402 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:27.701375    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:27.818053 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:27.818024    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:13:31.710583 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:31.710542    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj" podUID="edddd9d4-f442-4543-ae8b-f73319f9fbd4" containerName="registry" containerID="cri-o://a47df6247ad77a331be3d2e679a55e93682762d7e6fe80331cb7fccb0a31ede6" gracePeriod=30
Jul 25 01:13:31.725835 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:31.725816    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-559c56d4b8-fnljn"
Jul 25 01:13:31.830708 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:31.830684    2572 generic.go:356] "Generic (PLEG): container finished" podID="edddd9d4-f442-4543-ae8b-f73319f9fbd4" containerID="a47df6247ad77a331be3d2e679a55e93682762d7e6fe80331cb7fccb0a31ede6" exitCode=0
Jul 25 01:13:31.830810 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:31.830757    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj" event={"ID":"edddd9d4-f442-4543-ae8b-f73319f9fbd4","Type":"ContainerDied","Data":"a47df6247ad77a331be3d2e679a55e93682762d7e6fe80331cb7fccb0a31ede6"}
Jul 25 01:13:31.957607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:31.957585    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:13:32.129504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.129429    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-installation-pull-secrets\") pod \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") "
Jul 25 01:13:32.129504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.129470    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-trusted-ca\") pod \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") "
Jul 25 01:13:32.129504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.129496    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edddd9d4-f442-4543-ae8b-f73319f9fbd4-ca-trust-extracted\") pod \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") "
Jul 25 01:13:32.129757 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.129528    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-bound-sa-token\") pod \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") "
Jul 25 01:13:32.129757 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.129554    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ghqz\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-kube-api-access-6ghqz\") pod \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") "
Jul 25 01:13:32.129757 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.129582    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-certificates\") pod \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") "
Jul 25 01:13:32.129757 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.129610    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls\") pod \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") "
Jul 25 01:13:32.129757 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.129642    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-image-registry-private-configuration\") pod \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\" (UID: \"edddd9d4-f442-4543-ae8b-f73319f9fbd4\") "
Jul 25 01:13:32.130017 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.129932    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "edddd9d4-f442-4543-ae8b-f73319f9fbd4" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:13:32.130070 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.130025    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "edddd9d4-f442-4543-ae8b-f73319f9fbd4" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:13:32.131973 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.131946    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "edddd9d4-f442-4543-ae8b-f73319f9fbd4" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:13:32.132084 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.132050    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-kube-api-access-6ghqz" (OuterVolumeSpecName: "kube-api-access-6ghqz") pod "edddd9d4-f442-4543-ae8b-f73319f9fbd4" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4"). InnerVolumeSpecName "kube-api-access-6ghqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:13:32.132168 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.132135    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "edddd9d4-f442-4543-ae8b-f73319f9fbd4" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:13:32.132272 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.132247    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "edddd9d4-f442-4543-ae8b-f73319f9fbd4" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:13:32.132315 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.132271    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "edddd9d4-f442-4543-ae8b-f73319f9fbd4" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:13:32.138338 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.138305    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edddd9d4-f442-4543-ae8b-f73319f9fbd4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "edddd9d4-f442-4543-ae8b-f73319f9fbd4" (UID: "edddd9d4-f442-4543-ae8b-f73319f9fbd4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:13:32.230963 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.230937    2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-installation-pull-secrets\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:13:32.230963 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.230961    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-trusted-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:13:32.231115 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.230971    2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/edddd9d4-f442-4543-ae8b-f73319f9fbd4-ca-trust-extracted\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:13:32.231115 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.230981    2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-bound-sa-token\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:13:32.231115 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.230990    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ghqz\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-kube-api-access-6ghqz\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:13:32.231115 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.230998    2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-certificates\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:13:32.231115 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.231006    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/edddd9d4-f442-4543-ae8b-f73319f9fbd4-registry-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:13:32.231115 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.231014    2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/edddd9d4-f442-4543-ae8b-f73319f9fbd4-image-registry-private-configuration\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:13:32.836162 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.836129    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj" event={"ID":"edddd9d4-f442-4543-ae8b-f73319f9fbd4","Type":"ContainerDied","Data":"2f42c1170809cf019b21debb011a0647462c6b1b414170d0c04985fe91b72cda"}
Jul 25 01:13:32.836573 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.836162    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f58d88f64-52gnj"
Jul 25 01:13:32.836573 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.836222    2572 scope.go:117] "RemoveContainer" containerID="a47df6247ad77a331be3d2e679a55e93682762d7e6fe80331cb7fccb0a31ede6"
Jul 25 01:13:32.852984 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.852962    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f58d88f64-52gnj"]
Jul 25 01:13:32.859435 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:32.859413    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f58d88f64-52gnj"]
Jul 25 01:13:33.200685 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.200652    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-854469fbb-qs6b6"]
Jul 25 01:13:33.200934 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.200921    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edddd9d4-f442-4543-ae8b-f73319f9fbd4" containerName="registry"
Jul 25 01:13:33.200981 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.200936    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="edddd9d4-f442-4543-ae8b-f73319f9fbd4" containerName="registry"
Jul 25 01:13:33.201014 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.200989    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="edddd9d4-f442-4543-ae8b-f73319f9fbd4" containerName="registry"
Jul 25 01:13:33.208491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.208471    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.215136 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.215108    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854469fbb-qs6b6"]
Jul 25 01:13:33.215788 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.215624    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 25 01:13:33.340567 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.340539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-service-ca\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.340720 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.340574    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-oauth-config\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.340720 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.340611    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-oauth-serving-cert\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.340720 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.340675    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-serving-cert\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.340880 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.340736    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-console-config\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.340880 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.340785    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-trusted-ca-bundle\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.340880 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.340813    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xfkf\" (UniqueName: \"kubernetes.io/projected/b7e4b56a-0258-44ed-9303-f50307da92ea-kube-api-access-2xfkf\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.442128 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.442100    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-oauth-config\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.442257 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.442149    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-oauth-serving-cert\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.442257 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.442166    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-serving-cert\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.442399 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.442262    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-console-config\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.442399 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.442315    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-trusted-ca-bundle\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.442399 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.442351    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xfkf\" (UniqueName: \"kubernetes.io/projected/b7e4b56a-0258-44ed-9303-f50307da92ea-kube-api-access-2xfkf\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.442399 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.442386    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-service-ca\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.443034 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.442996    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-oauth-serving-cert\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.443034 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.443026    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-console-config\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.443157 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.443051    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-service-ca\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.443197 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.443180    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-trusted-ca-bundle\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.444559 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.444538    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-oauth-config\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.444657 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.444620    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-serving-cert\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.450239 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.450219    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xfkf\" (UniqueName: \"kubernetes.io/projected/b7e4b56a-0258-44ed-9303-f50307da92ea-kube-api-access-2xfkf\") pod \"console-854469fbb-qs6b6\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") " pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.518133 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.518081    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:33.666950 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.666923    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854469fbb-qs6b6"]
Jul 25 01:13:33.669284 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:13:33.669257    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e4b56a_0258_44ed_9303_f50307da92ea.slice/crio-93a9946b766b9becc24db17ea2736dbe523a9915cec6dffe41937ce645bc547c WatchSource:0}: Error finding container 93a9946b766b9becc24db17ea2736dbe523a9915cec6dffe41937ce645bc547c: Status 404 returned error can't find the container with id 93a9946b766b9becc24db17ea2736dbe523a9915cec6dffe41937ce645bc547c
Jul 25 01:13:33.842507 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.842423    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854469fbb-qs6b6" event={"ID":"b7e4b56a-0258-44ed-9303-f50307da92ea","Type":"ContainerStarted","Data":"648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6"}
Jul 25 01:13:33.842507 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.842468    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854469fbb-qs6b6" event={"ID":"b7e4b56a-0258-44ed-9303-f50307da92ea","Type":"ContainerStarted","Data":"93a9946b766b9becc24db17ea2736dbe523a9915cec6dffe41937ce645bc547c"}
Jul 25 01:13:33.866642 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:33.866594    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-854469fbb-qs6b6" podStartSLOduration=0.866574865 podStartE2EDuration="866.574865ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:13:33.865014638 +0000 UTC m=+97.974824046" watchObservedRunningTime="2026-07-25 01:13:33.866574865 +0000 UTC m=+97.976384149"
Jul 25 01:13:34.351897 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:34.351863    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edddd9d4-f442-4543-ae8b-f73319f9fbd4" path="/var/lib/kubelet/pods/edddd9d4-f442-4543-ae8b-f73319f9fbd4/volumes"
Jul 25 01:13:41.724232 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:41.724189    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qqdd4"
Jul 25 01:13:43.518878 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:43.518834    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:43.519311 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:43.518896    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:43.523862 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:43.523839    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:43.882086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:43.882012    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:13:43.923212 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:43.923177    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-654f5bcb7b-788cl"]
Jul 25 01:13:46.234564 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:13:46.234529    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x62hp_cc0d00f8-2138-4cf2-98bf-d1898e41ddd1/serve-healthcheck-canary/0.log"
Jul 25 01:14:08.943061 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:08.943018    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-654f5bcb7b-788cl" podUID="42b100c8-65dc-481b-9e38-81221ca55647" containerName="console" containerID="cri-o://8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e" gracePeriod=15
Jul 25 01:14:09.219634 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.219570    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:14:09.311566 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.311531    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-oauth-config\") pod \"42b100c8-65dc-481b-9e38-81221ca55647\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") "
Jul 25 01:14:09.311737 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.311595    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-serving-cert\") pod \"42b100c8-65dc-481b-9e38-81221ca55647\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") "
Jul 25 01:14:09.311737 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.311616    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-console-config\") pod \"42b100c8-65dc-481b-9e38-81221ca55647\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") "
Jul 25 01:14:09.311737 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.311634    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-service-ca\") pod \"42b100c8-65dc-481b-9e38-81221ca55647\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") "
Jul 25 01:14:09.311737 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.311679    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-oauth-serving-cert\") pod \"42b100c8-65dc-481b-9e38-81221ca55647\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") "
Jul 25 01:14:09.311941 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.311787    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szgh8\" (UniqueName: \"kubernetes.io/projected/42b100c8-65dc-481b-9e38-81221ca55647-kube-api-access-szgh8\") pod \"42b100c8-65dc-481b-9e38-81221ca55647\" (UID: \"42b100c8-65dc-481b-9e38-81221ca55647\") "
Jul 25 01:14:09.312072 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.312024    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "42b100c8-65dc-481b-9e38-81221ca55647" (UID: "42b100c8-65dc-481b-9e38-81221ca55647"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:14:09.312072 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.312018    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-service-ca" (OuterVolumeSpecName: "service-ca") pod "42b100c8-65dc-481b-9e38-81221ca55647" (UID: "42b100c8-65dc-481b-9e38-81221ca55647"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:14:09.312377 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.312263    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-console-config" (OuterVolumeSpecName: "console-config") pod "42b100c8-65dc-481b-9e38-81221ca55647" (UID: "42b100c8-65dc-481b-9e38-81221ca55647"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:14:09.314432 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.314392    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "42b100c8-65dc-481b-9e38-81221ca55647" (UID: "42b100c8-65dc-481b-9e38-81221ca55647"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:14:09.314432 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.314405    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b100c8-65dc-481b-9e38-81221ca55647-kube-api-access-szgh8" (OuterVolumeSpecName: "kube-api-access-szgh8") pod "42b100c8-65dc-481b-9e38-81221ca55647" (UID: "42b100c8-65dc-481b-9e38-81221ca55647"). InnerVolumeSpecName "kube-api-access-szgh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:14:09.314802 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.314775    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "42b100c8-65dc-481b-9e38-81221ca55647" (UID: "42b100c8-65dc-481b-9e38-81221ca55647"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:14:09.412654 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.412623    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-oauth-config\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:14:09.412654 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.412653    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42b100c8-65dc-481b-9e38-81221ca55647-console-serving-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:14:09.412860 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.412663    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-console-config\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:14:09.412860 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.412672    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:14:09.412860 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.412680    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42b100c8-65dc-481b-9e38-81221ca55647-oauth-serving-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:14:09.412860 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.412689    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szgh8\" (UniqueName: \"kubernetes.io/projected/42b100c8-65dc-481b-9e38-81221ca55647-kube-api-access-szgh8\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:14:09.973400 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.973369    2572 generic.go:356] "Generic (PLEG): container finished" podID="42b100c8-65dc-481b-9e38-81221ca55647" containerID="8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e" exitCode=0
Jul 25 01:14:09.973819 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.973414    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-654f5bcb7b-788cl"
Jul 25 01:14:09.973819 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.973427    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654f5bcb7b-788cl" event={"ID":"42b100c8-65dc-481b-9e38-81221ca55647","Type":"ContainerDied","Data":"8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e"}
Jul 25 01:14:09.973819 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.973468    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-654f5bcb7b-788cl" event={"ID":"42b100c8-65dc-481b-9e38-81221ca55647","Type":"ContainerDied","Data":"b0f69d5d1dfa65489fc80880cb496e225c021fac5047572b2f537434cbd1f773"}
Jul 25 01:14:09.973819 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.973487    2572 scope.go:117] "RemoveContainer" containerID="8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e"
Jul 25 01:14:09.982095 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.982076    2572 scope.go:117] "RemoveContainer" containerID="8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e"
Jul 25 01:14:09.982333 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:14:09.982301    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e\": container with ID starting with 8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e not found: ID does not exist" containerID="8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e"
Jul 25 01:14:09.982406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.982341    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e"} err="failed to get container status \"8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e\": rpc error: code = NotFound desc = could not find container \"8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e\": container with ID starting with 8178a3260d750fd7f2d3e7cf7dfebc09d58f190b2da79cea8de159f3a91d919e not found: ID does not exist"
Jul 25 01:14:09.993556 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.993535    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-654f5bcb7b-788cl"]
Jul 25 01:14:09.997237 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:09.997216    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-654f5bcb7b-788cl"]
Jul 25 01:14:10.352178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:10.352098    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b100c8-65dc-481b-9e38-81221ca55647" path="/var/lib/kubelet/pods/42b100c8-65dc-481b-9e38-81221ca55647/volumes"
Jul 25 01:14:51.556422 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.556386    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-798589c9d5-kklwr"]
Jul 25 01:14:51.556912 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.556662    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b100c8-65dc-481b-9e38-81221ca55647" containerName="console"
Jul 25 01:14:51.556912 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.556672    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b100c8-65dc-481b-9e38-81221ca55647" containerName="console"
Jul 25 01:14:51.556912 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.556739    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b100c8-65dc-481b-9e38-81221ca55647" containerName="console"
Jul 25 01:14:51.559786 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.559766    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.572657 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.572633    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-798589c9d5-kklwr"]
Jul 25 01:14:51.731412 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.731377    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7wrl\" (UniqueName: \"kubernetes.io/projected/311fa83a-a763-4098-97e7-eb16804e9b76-kube-api-access-k7wrl\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.731580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.731424    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-oauth-config\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.731580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.731440    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-service-ca\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.731580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.731479    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-trusted-ca-bundle\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.731580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.731496    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-console-config\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.731580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.731517    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-serving-cert\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.731580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.731533    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-oauth-serving-cert\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.832176 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.832116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-oauth-config\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.832176 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.832148    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-service-ca\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.832176 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.832174    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-trusted-ca-bundle\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.832425 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.832197    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-console-config\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.832425 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.832218    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-serving-cert\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.832425 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.832237    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-oauth-serving-cert\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.832425 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.832284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7wrl\" (UniqueName: \"kubernetes.io/projected/311fa83a-a763-4098-97e7-eb16804e9b76-kube-api-access-k7wrl\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.833013 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.832983    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-console-config\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.833013 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.832999    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-oauth-serving-cert\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.833136 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.832983    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-service-ca\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.833172 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.833147    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-trusted-ca-bundle\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.834641 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.834613    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-oauth-config\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.834735 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.834722    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-serving-cert\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.840230 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.840208    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7wrl\" (UniqueName: \"kubernetes.io/projected/311fa83a-a763-4098-97e7-eb16804e9b76-kube-api-access-k7wrl\") pod \"console-798589c9d5-kklwr\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") " pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.868907 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.868887    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:14:51.995953 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:51.995929    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-798589c9d5-kklwr"]
Jul 25 01:14:51.997897 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:14:51.997864    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311fa83a_a763_4098_97e7_eb16804e9b76.slice/crio-fc57a7bb8591e0a82924665c5b3ad442752d0b0e7f71d8412a5abf84a647a8ae WatchSource:0}: Error finding container fc57a7bb8591e0a82924665c5b3ad442752d0b0e7f71d8412a5abf84a647a8ae: Status 404 returned error can't find the container with id fc57a7bb8591e0a82924665c5b3ad442752d0b0e7f71d8412a5abf84a647a8ae
Jul 25 01:14:52.120104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:52.119983    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798589c9d5-kklwr" event={"ID":"311fa83a-a763-4098-97e7-eb16804e9b76","Type":"ContainerStarted","Data":"90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027"}
Jul 25 01:14:52.120104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:52.120044    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798589c9d5-kklwr" event={"ID":"311fa83a-a763-4098-97e7-eb16804e9b76","Type":"ContainerStarted","Data":"fc57a7bb8591e0a82924665c5b3ad442752d0b0e7f71d8412a5abf84a647a8ae"}
Jul 25 01:14:52.133948 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:14:52.133894    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-798589c9d5-kklwr" podStartSLOduration=1.133875619 podStartE2EDuration="1.133875619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:14:52.133305305 +0000 UTC m=+176.243114614" watchObservedRunningTime="2026-07-25 01:14:52.133875619 +0000 UTC m=+176.243684904"
Jul 25 01:15:01.869863 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:01.869823    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:15:01.870284 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:01.869905    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:15:01.874813 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:01.874794    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:15:02.157766 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:02.157737    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:15:02.192802 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:02.192762    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854469fbb-qs6b6"]
Jul 25 01:15:27.212228 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.212184    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-854469fbb-qs6b6" podUID="b7e4b56a-0258-44ed-9303-f50307da92ea" containerName="console" containerID="cri-o://648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6" gracePeriod=15
Jul 25 01:15:27.458611 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.458588    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:15:27.584347 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.584236    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-oauth-serving-cert\") pod \"b7e4b56a-0258-44ed-9303-f50307da92ea\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") "
Jul 25 01:15:27.584347 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.584286    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xfkf\" (UniqueName: \"kubernetes.io/projected/b7e4b56a-0258-44ed-9303-f50307da92ea-kube-api-access-2xfkf\") pod \"b7e4b56a-0258-44ed-9303-f50307da92ea\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") "
Jul 25 01:15:27.584573 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.584355    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-oauth-config\") pod \"b7e4b56a-0258-44ed-9303-f50307da92ea\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") "
Jul 25 01:15:27.584573 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.584424    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-service-ca\") pod \"b7e4b56a-0258-44ed-9303-f50307da92ea\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") "
Jul 25 01:15:27.584573 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.584462    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-console-config\") pod \"b7e4b56a-0258-44ed-9303-f50307da92ea\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") "
Jul 25 01:15:27.584573 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.584489    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-trusted-ca-bundle\") pod \"b7e4b56a-0258-44ed-9303-f50307da92ea\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") "
Jul 25 01:15:27.584573 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.584530    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-serving-cert\") pod \"b7e4b56a-0258-44ed-9303-f50307da92ea\" (UID: \"b7e4b56a-0258-44ed-9303-f50307da92ea\") "
Jul 25 01:15:27.584945 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.584896    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-console-config" (OuterVolumeSpecName: "console-config") pod "b7e4b56a-0258-44ed-9303-f50307da92ea" (UID: "b7e4b56a-0258-44ed-9303-f50307da92ea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:15:27.584945 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.584905    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7e4b56a-0258-44ed-9303-f50307da92ea" (UID: "b7e4b56a-0258-44ed-9303-f50307da92ea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:15:27.585108 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.584916    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b7e4b56a-0258-44ed-9303-f50307da92ea" (UID: "b7e4b56a-0258-44ed-9303-f50307da92ea"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:15:27.585108 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.584985    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7e4b56a-0258-44ed-9303-f50307da92ea" (UID: "b7e4b56a-0258-44ed-9303-f50307da92ea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:15:27.586581 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.586558    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7e4b56a-0258-44ed-9303-f50307da92ea" (UID: "b7e4b56a-0258-44ed-9303-f50307da92ea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:15:27.586904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.586880    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7e4b56a-0258-44ed-9303-f50307da92ea" (UID: "b7e4b56a-0258-44ed-9303-f50307da92ea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:15:27.586904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.586892    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e4b56a-0258-44ed-9303-f50307da92ea-kube-api-access-2xfkf" (OuterVolumeSpecName: "kube-api-access-2xfkf") pod "b7e4b56a-0258-44ed-9303-f50307da92ea" (UID: "b7e4b56a-0258-44ed-9303-f50307da92ea"). InnerVolumeSpecName "kube-api-access-2xfkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:15:27.685907 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.685874    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-oauth-config\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:15:27.685907 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.685899    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:15:27.685907 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.685909    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-console-config\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:15:27.686097 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.685916    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-trusted-ca-bundle\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:15:27.686097 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.685926    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7e4b56a-0258-44ed-9303-f50307da92ea-console-serving-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:15:27.686097 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.685934    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7e4b56a-0258-44ed-9303-f50307da92ea-oauth-serving-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:15:27.686097 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:27.685942    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xfkf\" (UniqueName: \"kubernetes.io/projected/b7e4b56a-0258-44ed-9303-f50307da92ea-kube-api-access-2xfkf\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:15:28.246773 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:28.246736    2572 generic.go:356] "Generic (PLEG): container finished" podID="b7e4b56a-0258-44ed-9303-f50307da92ea" containerID="648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6" exitCode=0
Jul 25 01:15:28.247129 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:28.246785    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854469fbb-qs6b6"
Jul 25 01:15:28.247129 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:28.246786    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854469fbb-qs6b6" event={"ID":"b7e4b56a-0258-44ed-9303-f50307da92ea","Type":"ContainerDied","Data":"648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6"}
Jul 25 01:15:28.247129 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:28.246836    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854469fbb-qs6b6" event={"ID":"b7e4b56a-0258-44ed-9303-f50307da92ea","Type":"ContainerDied","Data":"93a9946b766b9becc24db17ea2736dbe523a9915cec6dffe41937ce645bc547c"}
Jul 25 01:15:28.247129 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:28.246866    2572 scope.go:117] "RemoveContainer" containerID="648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6"
Jul 25 01:15:28.254952 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:28.254923    2572 scope.go:117] "RemoveContainer" containerID="648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6"
Jul 25 01:15:28.255212 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:15:28.255188    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6\": container with ID starting with 648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6 not found: ID does not exist" containerID="648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6"
Jul 25 01:15:28.255264 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:28.255225    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6"} err="failed to get container status \"648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6\": rpc error: code = NotFound desc = could not find container \"648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6\": container with ID starting with 648bff992b554db4530ac0af8fe3b98ebdaea7ea6fe9c267cc61081108a348e6 not found: ID does not exist"
Jul 25 01:15:28.266584 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:28.266564    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854469fbb-qs6b6"]
Jul 25 01:15:28.270215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:28.270192    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-854469fbb-qs6b6"]
Jul 25 01:15:28.351979 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:15:28.351948    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e4b56a-0258-44ed-9303-f50307da92ea" path="/var/lib/kubelet/pods/b7e4b56a-0258-44ed-9303-f50307da92ea/volumes"
Jul 25 01:16:12.270056 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.270021    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76789559f6-p87vj"]
Jul 25 01:16:12.270528 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.270299    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e4b56a-0258-44ed-9303-f50307da92ea" containerName="console"
Jul 25 01:16:12.270528 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.270310    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e4b56a-0258-44ed-9303-f50307da92ea" containerName="console"
Jul 25 01:16:12.270528 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.270385    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e4b56a-0258-44ed-9303-f50307da92ea" containerName="console"
Jul 25 01:16:12.274335 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.274303    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.284137 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.284113    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76789559f6-p87vj"]
Jul 25 01:16:12.404703 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.404677    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-trusted-ca-bundle\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.404876 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.404708    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-serving-cert\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.404876 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.404734    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-service-ca\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.404876 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.404749    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-oauth-serving-cert\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.404876 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.404815    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-oauth-config\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.405073 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.404877    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpl6\" (UniqueName: \"kubernetes.io/projected/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-kube-api-access-cwpl6\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.405073 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.404951    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-config\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.506002 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.505969    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-oauth-config\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.506180 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.506012    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpl6\" (UniqueName: \"kubernetes.io/projected/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-kube-api-access-cwpl6\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.506180 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.506135    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-config\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.506311 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.506181    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-trusted-ca-bundle\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.506311 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.506223    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-serving-cert\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.506311 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.506287    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-service-ca\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.506541 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.506311    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-oauth-serving-cert\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.506985 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.506960    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-oauth-serving-cert\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.507115 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.507044    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-config\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.507115 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.507067    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-trusted-ca-bundle\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.507233 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.507132    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-service-ca\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.508672 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.508647    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-oauth-config\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.508780 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.508705    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-serving-cert\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.513941 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.513918    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpl6\" (UniqueName: \"kubernetes.io/projected/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-kube-api-access-cwpl6\") pod \"console-76789559f6-p87vj\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") " pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.582872 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.582801    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:12.709704 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:12.709663    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76789559f6-p87vj"]
Jul 25 01:16:12.714425 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:16:12.714398    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac9b1d5_1cfa_48f2_8763_de945ac9ce34.slice/crio-163115bcdcd3085fbc567accf36b2edf0d651558b84521e96235e70885da3248 WatchSource:0}: Error finding container 163115bcdcd3085fbc567accf36b2edf0d651558b84521e96235e70885da3248: Status 404 returned error can't find the container with id 163115bcdcd3085fbc567accf36b2edf0d651558b84521e96235e70885da3248
Jul 25 01:16:13.400417 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:13.400384    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76789559f6-p87vj" event={"ID":"fac9b1d5-1cfa-48f2-8763-de945ac9ce34","Type":"ContainerStarted","Data":"4335090e58429957bd30f3bb402df1c6b77e6c0d97d50a80680381134d2a7ef9"}
Jul 25 01:16:13.400790 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:13.400434    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76789559f6-p87vj" event={"ID":"fac9b1d5-1cfa-48f2-8763-de945ac9ce34","Type":"ContainerStarted","Data":"163115bcdcd3085fbc567accf36b2edf0d651558b84521e96235e70885da3248"}
Jul 25 01:16:13.417701 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:13.417652    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76789559f6-p87vj" podStartSLOduration=1.4176377 podStartE2EDuration="1.4176377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:16:13.416639453 +0000 UTC m=+257.526448737" watchObservedRunningTime="2026-07-25 01:16:13.4176377 +0000 UTC m=+257.527446983"
Jul 25 01:16:22.583656 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:22.583582    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:22.583656 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:22.583640    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:22.588090 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:22.588069    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:23.438678 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:23.438649    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:16:23.482762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:23.482728    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-798589c9d5-kklwr"]
Jul 25 01:16:48.503317 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.503258    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-798589c9d5-kklwr" podUID="311fa83a-a763-4098-97e7-eb16804e9b76" containerName="console" containerID="cri-o://90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027" gracePeriod=15
Jul 25 01:16:48.749190 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.749165    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:16:48.867074 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.866985    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-serving-cert\") pod \"311fa83a-a763-4098-97e7-eb16804e9b76\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") "
Jul 25 01:16:48.867074 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.867047    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-oauth-config\") pod \"311fa83a-a763-4098-97e7-eb16804e9b76\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") "
Jul 25 01:16:48.867283 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.867080    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-trusted-ca-bundle\") pod \"311fa83a-a763-4098-97e7-eb16804e9b76\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") "
Jul 25 01:16:48.867283 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.867102    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-oauth-serving-cert\") pod \"311fa83a-a763-4098-97e7-eb16804e9b76\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") "
Jul 25 01:16:48.867283 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.867124    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-service-ca\") pod \"311fa83a-a763-4098-97e7-eb16804e9b76\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") "
Jul 25 01:16:48.867283 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.867177    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-console-config\") pod \"311fa83a-a763-4098-97e7-eb16804e9b76\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") "
Jul 25 01:16:48.867283 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.867212    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7wrl\" (UniqueName: \"kubernetes.io/projected/311fa83a-a763-4098-97e7-eb16804e9b76-kube-api-access-k7wrl\") pod \"311fa83a-a763-4098-97e7-eb16804e9b76\" (UID: \"311fa83a-a763-4098-97e7-eb16804e9b76\") "
Jul 25 01:16:48.867607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.867581    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "311fa83a-a763-4098-97e7-eb16804e9b76" (UID: "311fa83a-a763-4098-97e7-eb16804e9b76"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:16:48.867687 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.867668    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-service-ca" (OuterVolumeSpecName: "service-ca") pod "311fa83a-a763-4098-97e7-eb16804e9b76" (UID: "311fa83a-a763-4098-97e7-eb16804e9b76"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:16:48.867735 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.867665    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-console-config" (OuterVolumeSpecName: "console-config") pod "311fa83a-a763-4098-97e7-eb16804e9b76" (UID: "311fa83a-a763-4098-97e7-eb16804e9b76"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:16:48.867831 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.867812    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "311fa83a-a763-4098-97e7-eb16804e9b76" (UID: "311fa83a-a763-4098-97e7-eb16804e9b76"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:16:48.869306 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.869274    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311fa83a-a763-4098-97e7-eb16804e9b76-kube-api-access-k7wrl" (OuterVolumeSpecName: "kube-api-access-k7wrl") pod "311fa83a-a763-4098-97e7-eb16804e9b76" (UID: "311fa83a-a763-4098-97e7-eb16804e9b76"). InnerVolumeSpecName "kube-api-access-k7wrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:16:48.869666 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.869646    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "311fa83a-a763-4098-97e7-eb16804e9b76" (UID: "311fa83a-a763-4098-97e7-eb16804e9b76"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:16:48.869745 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.869679    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "311fa83a-a763-4098-97e7-eb16804e9b76" (UID: "311fa83a-a763-4098-97e7-eb16804e9b76"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:16:48.968209 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.968176    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-oauth-config\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:16:48.968209 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.968202    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-trusted-ca-bundle\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:16:48.968209 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.968212    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-oauth-serving-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:16:48.968443 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.968220    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:16:48.968443 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.968228    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311fa83a-a763-4098-97e7-eb16804e9b76-console-config\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:16:48.968443 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.968236    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7wrl\" (UniqueName: \"kubernetes.io/projected/311fa83a-a763-4098-97e7-eb16804e9b76-kube-api-access-k7wrl\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:16:48.968443 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:48.968244    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311fa83a-a763-4098-97e7-eb16804e9b76-console-serving-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:16:49.524086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:49.524054    2572 generic.go:356] "Generic (PLEG): container finished" podID="311fa83a-a763-4098-97e7-eb16804e9b76" containerID="90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027" exitCode=0
Jul 25 01:16:49.524504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:49.524094    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-798589c9d5-kklwr"
Jul 25 01:16:49.524504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:49.524111    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798589c9d5-kklwr" event={"ID":"311fa83a-a763-4098-97e7-eb16804e9b76","Type":"ContainerDied","Data":"90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027"}
Jul 25 01:16:49.524504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:49.524159    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-798589c9d5-kklwr" event={"ID":"311fa83a-a763-4098-97e7-eb16804e9b76","Type":"ContainerDied","Data":"fc57a7bb8591e0a82924665c5b3ad442752d0b0e7f71d8412a5abf84a647a8ae"}
Jul 25 01:16:49.524504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:49.524180    2572 scope.go:117] "RemoveContainer" containerID="90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027"
Jul 25 01:16:49.532388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:49.532370    2572 scope.go:117] "RemoveContainer" containerID="90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027"
Jul 25 01:16:49.532641 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:16:49.532624    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027\": container with ID starting with 90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027 not found: ID does not exist" containerID="90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027"
Jul 25 01:16:49.532685 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:49.532651    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027"} err="failed to get container status \"90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027\": rpc error: code = NotFound desc = could not find container \"90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027\": container with ID starting with 90edd0152ef1ac923139e286dcc5f5a58015158a602c0e726066c41490aa2027 not found: ID does not exist"
Jul 25 01:16:49.545115 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:49.545092    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-798589c9d5-kklwr"]
Jul 25 01:16:49.553419 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:49.553390    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-798589c9d5-kklwr"]
Jul 25 01:16:50.352795 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:50.352759    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311fa83a-a763-4098-97e7-eb16804e9b76" path="/var/lib/kubelet/pods/311fa83a-a763-4098-97e7-eb16804e9b76/volumes"
Jul 25 01:16:56.270756 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:16:56.270728    2572 kubelet.go:1648] "Image garbage collection succeeded"
Jul 25 01:17:11.407981 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.407948    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"]
Jul 25 01:17:11.410353 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.408233    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311fa83a-a763-4098-97e7-eb16804e9b76" containerName="console"
Jul 25 01:17:11.410353 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.408246    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="311fa83a-a763-4098-97e7-eb16804e9b76" containerName="console"
Jul 25 01:17:11.410353 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.408314    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="311fa83a-a763-4098-97e7-eb16804e9b76" containerName="console"
Jul 25 01:17:11.411167 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.411151    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:11.413565 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.413543    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-88vfc\""
Jul 25 01:17:11.413948 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.413931    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 25 01:17:11.414436 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.414421    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 25 01:17:11.420580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.420557    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"]
Jul 25 01:17:11.528726 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.528699    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:11.528871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.528742    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82b5l\" (UniqueName: \"kubernetes.io/projected/d3390fa4-e72f-486c-8f1b-694b5143aa2f-kube-api-access-82b5l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:11.528871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.528769    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:11.629102 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.629070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82b5l\" (UniqueName: \"kubernetes.io/projected/d3390fa4-e72f-486c-8f1b-694b5143aa2f-kube-api-access-82b5l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:11.629251 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.629116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:11.629251 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.629176    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:11.629531 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.629508    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:11.629531 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.629521    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:11.636812 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.636792    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82b5l\" (UniqueName: \"kubernetes.io/projected/d3390fa4-e72f-486c-8f1b-694b5143aa2f-kube-api-access-82b5l\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:11.719976 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.719910    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:11.842407 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.842383    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"]
Jul 25 01:17:11.845683 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:17:11.845655    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3390fa4_e72f_486c_8f1b_694b5143aa2f.slice/crio-f7b5abe78adcd0c15cf69c798cc9ff716fabbea4fa74a17c41d2b15b3d9f5a0f WatchSource:0}: Error finding container f7b5abe78adcd0c15cf69c798cc9ff716fabbea4fa74a17c41d2b15b3d9f5a0f: Status 404 returned error can't find the container with id f7b5abe78adcd0c15cf69c798cc9ff716fabbea4fa74a17c41d2b15b3d9f5a0f
Jul 25 01:17:11.850356 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:11.848258    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 25 01:17:12.603980 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:12.603943    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf" event={"ID":"d3390fa4-e72f-486c-8f1b-694b5143aa2f","Type":"ContainerStarted","Data":"f7b5abe78adcd0c15cf69c798cc9ff716fabbea4fa74a17c41d2b15b3d9f5a0f"}
Jul 25 01:17:18.628512 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:18.628472    2572 generic.go:356] "Generic (PLEG): container finished" podID="d3390fa4-e72f-486c-8f1b-694b5143aa2f" containerID="8155f202bbd262099071019a29070e4b14aa4a74ba20f558e7a435901f1b954c" exitCode=0
Jul 25 01:17:18.628950 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:18.628561    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf" event={"ID":"d3390fa4-e72f-486c-8f1b-694b5143aa2f","Type":"ContainerDied","Data":"8155f202bbd262099071019a29070e4b14aa4a74ba20f558e7a435901f1b954c"}
Jul 25 01:17:21.644443 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:21.644405    2572 generic.go:356] "Generic (PLEG): container finished" podID="d3390fa4-e72f-486c-8f1b-694b5143aa2f" containerID="e22baa9efa33a3d0b6e4e6ee52a7c9fdc74feba68f3743bd8f10a12b5f65d2ca" exitCode=0
Jul 25 01:17:21.644841 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:21.644471    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf" event={"ID":"d3390fa4-e72f-486c-8f1b-694b5143aa2f","Type":"ContainerDied","Data":"e22baa9efa33a3d0b6e4e6ee52a7c9fdc74feba68f3743bd8f10a12b5f65d2ca"}
Jul 25 01:17:28.674856 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:28.674820    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf" event={"ID":"d3390fa4-e72f-486c-8f1b-694b5143aa2f","Type":"ContainerStarted","Data":"da7d8ed0712fff6171a12faa52e97c0bd4e54d6633b82c1af892b92cc80e3e15"}
Jul 25 01:17:28.695144 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:28.695102    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf" podStartSLOduration=2.833783702 podStartE2EDuration="17.695088457s" totalImagesPullingTime="14.861304755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:17:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:17:28.69435202 +0000 UTC m=+332.804161303" watchObservedRunningTime="2026-07-25 01:17:28.695088457 +0000 UTC m=+332.804897740"
Jul 25 01:17:29.680120 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:29.680085    2572 generic.go:356] "Generic (PLEG): container finished" podID="d3390fa4-e72f-486c-8f1b-694b5143aa2f" containerID="da7d8ed0712fff6171a12faa52e97c0bd4e54d6633b82c1af892b92cc80e3e15" exitCode=0
Jul 25 01:17:29.680510 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:29.680133    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf" event={"ID":"d3390fa4-e72f-486c-8f1b-694b5143aa2f","Type":"ContainerDied","Data":"da7d8ed0712fff6171a12faa52e97c0bd4e54d6633b82c1af892b92cc80e3e15"}
Jul 25 01:17:30.836864 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:30.836836    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:30.889050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:30.889024    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-util\") pod \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") "
Jul 25 01:17:30.889173 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:30.889075    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82b5l\" (UniqueName: \"kubernetes.io/projected/d3390fa4-e72f-486c-8f1b-694b5143aa2f-kube-api-access-82b5l\") pod \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") "
Jul 25 01:17:30.889173 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:30.889096    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-bundle\") pod \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\" (UID: \"d3390fa4-e72f-486c-8f1b-694b5143aa2f\") "
Jul 25 01:17:30.889822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:30.889797    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-bundle" (OuterVolumeSpecName: "bundle") pod "d3390fa4-e72f-486c-8f1b-694b5143aa2f" (UID: "d3390fa4-e72f-486c-8f1b-694b5143aa2f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:17:30.891313 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:30.891291    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3390fa4-e72f-486c-8f1b-694b5143aa2f-kube-api-access-82b5l" (OuterVolumeSpecName: "kube-api-access-82b5l") pod "d3390fa4-e72f-486c-8f1b-694b5143aa2f" (UID: "d3390fa4-e72f-486c-8f1b-694b5143aa2f"). InnerVolumeSpecName "kube-api-access-82b5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:17:30.894117 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:30.894094    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-util" (OuterVolumeSpecName: "util") pod "d3390fa4-e72f-486c-8f1b-694b5143aa2f" (UID: "d3390fa4-e72f-486c-8f1b-694b5143aa2f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:17:30.989812 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:30.989744    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-bundle\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:17:30.989812 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:30.989770    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3390fa4-e72f-486c-8f1b-694b5143aa2f-util\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:17:30.989812 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:30.989780    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82b5l\" (UniqueName: \"kubernetes.io/projected/d3390fa4-e72f-486c-8f1b-694b5143aa2f-kube-api-access-82b5l\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:17:31.689823 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:31.689781    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf" event={"ID":"d3390fa4-e72f-486c-8f1b-694b5143aa2f","Type":"ContainerDied","Data":"f7b5abe78adcd0c15cf69c798cc9ff716fabbea4fa74a17c41d2b15b3d9f5a0f"}
Jul 25 01:17:31.689823 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:31.689807    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540hchvf"
Jul 25 01:17:31.690089 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:31.689841    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7b5abe78adcd0c15cf69c798cc9ff716fabbea4fa74a17c41d2b15b3d9f5a0f"
Jul 25 01:17:43.414006 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.413970    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"]
Jul 25 01:17:43.414401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.414224    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3390fa4-e72f-486c-8f1b-694b5143aa2f" containerName="pull"
Jul 25 01:17:43.414401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.414234    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3390fa4-e72f-486c-8f1b-694b5143aa2f" containerName="pull"
Jul 25 01:17:43.414401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.414242    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3390fa4-e72f-486c-8f1b-694b5143aa2f" containerName="util"
Jul 25 01:17:43.414401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.414247    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3390fa4-e72f-486c-8f1b-694b5143aa2f" containerName="util"
Jul 25 01:17:43.414401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.414261    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3390fa4-e72f-486c-8f1b-694b5143aa2f" containerName="extract"
Jul 25 01:17:43.414401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.414267    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3390fa4-e72f-486c-8f1b-694b5143aa2f" containerName="extract"
Jul 25 01:17:43.414401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.414315    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3390fa4-e72f-486c-8f1b-694b5143aa2f" containerName="extract"
Jul 25 01:17:43.417510 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.417491    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:43.423677 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.423454    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-88vfc\""
Jul 25 01:17:43.423677 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.423478    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 25 01:17:43.423677 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.423498    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 25 01:17:43.427977 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.427956    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"]
Jul 25 01:17:43.477149 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.477120    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:43.477270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.477163    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77rw2\" (UniqueName: \"kubernetes.io/projected/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-kube-api-access-77rw2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:43.477270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.477187    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:43.578086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.578057    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77rw2\" (UniqueName: \"kubernetes.io/projected/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-kube-api-access-77rw2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:43.578086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.578089    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:43.578270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.578132    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:43.578484 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.578468    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:43.578520 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.578502    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:43.587656 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.587629    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77rw2\" (UniqueName: \"kubernetes.io/projected/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-kube-api-access-77rw2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:43.726159 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.726070    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:43.835879 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.835850    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5mqqf"]
Jul 25 01:17:43.838051 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.838030    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf"
Jul 25 01:17:43.840423 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.840402    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 25 01:17:43.840423 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.840415    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rl52t\""
Jul 25 01:17:43.840565 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.840442    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 25 01:17:43.846347 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.846310    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5mqqf"]
Jul 25 01:17:43.851378 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.851356    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"]
Jul 25 01:17:43.856372 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:17:43.856345    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4f9cc8_68d5_4f7b_8d85_2d04714713b0.slice/crio-126d8dcbfa8bb70169ffd0600bf3df6d7695a4e6d7f10a651aa9c2378b717dcb WatchSource:0}: Error finding container 126d8dcbfa8bb70169ffd0600bf3df6d7695a4e6d7f10a651aa9c2378b717dcb: Status 404 returned error can't find the container with id 126d8dcbfa8bb70169ffd0600bf3df6d7695a4e6d7f10a651aa9c2378b717dcb
Jul 25 01:17:43.879951 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.879928    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5hrq\" (UniqueName: \"kubernetes.io/projected/4714b753-bcfd-4c76-91b5-650a94a6df3e-kube-api-access-z5hrq\") pod \"cert-manager-webhook-6486ffd997-5mqqf\" (UID: \"4714b753-bcfd-4c76-91b5-650a94a6df3e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf"
Jul 25 01:17:43.880035 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.879980    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4714b753-bcfd-4c76-91b5-650a94a6df3e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5mqqf\" (UID: \"4714b753-bcfd-4c76-91b5-650a94a6df3e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf"
Jul 25 01:17:43.981239 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.981179    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4714b753-bcfd-4c76-91b5-650a94a6df3e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5mqqf\" (UID: \"4714b753-bcfd-4c76-91b5-650a94a6df3e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf"
Jul 25 01:17:43.981353 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.981246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5hrq\" (UniqueName: \"kubernetes.io/projected/4714b753-bcfd-4c76-91b5-650a94a6df3e-kube-api-access-z5hrq\") pod \"cert-manager-webhook-6486ffd997-5mqqf\" (UID: \"4714b753-bcfd-4c76-91b5-650a94a6df3e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf"
Jul 25 01:17:43.988795 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.988774    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5hrq\" (UniqueName: \"kubernetes.io/projected/4714b753-bcfd-4c76-91b5-650a94a6df3e-kube-api-access-z5hrq\") pod \"cert-manager-webhook-6486ffd997-5mqqf\" (UID: \"4714b753-bcfd-4c76-91b5-650a94a6df3e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf"
Jul 25 01:17:43.988873 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:43.988794    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4714b753-bcfd-4c76-91b5-650a94a6df3e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-5mqqf\" (UID: \"4714b753-bcfd-4c76-91b5-650a94a6df3e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf"
Jul 25 01:17:44.157021 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:44.156982    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf"
Jul 25 01:17:44.280577 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:44.280548    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-5mqqf"]
Jul 25 01:17:44.283906 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:17:44.283881    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4714b753_bcfd_4c76_91b5_650a94a6df3e.slice/crio-559c072c2806afe2c6f9d60f423c58176e12063992ed862dbb250718187fa8d7 WatchSource:0}: Error finding container 559c072c2806afe2c6f9d60f423c58176e12063992ed862dbb250718187fa8d7: Status 404 returned error can't find the container with id 559c072c2806afe2c6f9d60f423c58176e12063992ed862dbb250718187fa8d7
Jul 25 01:17:44.737857 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:44.737823    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf" event={"ID":"4714b753-bcfd-4c76-91b5-650a94a6df3e","Type":"ContainerStarted","Data":"559c072c2806afe2c6f9d60f423c58176e12063992ed862dbb250718187fa8d7"}
Jul 25 01:17:44.739245 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:44.739217    2572 generic.go:356] "Generic (PLEG): container finished" podID="6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" containerID="2100e97dfb8b46d36107be3bf4c84b96a295957ec891666f9381c0786b39df84" exitCode=0
Jul 25 01:17:44.739382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:44.739249    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg" event={"ID":"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0","Type":"ContainerDied","Data":"2100e97dfb8b46d36107be3bf4c84b96a295957ec891666f9381c0786b39df84"}
Jul 25 01:17:44.739382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:44.739283    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg" event={"ID":"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0","Type":"ContainerStarted","Data":"126d8dcbfa8bb70169ffd0600bf3df6d7695a4e6d7f10a651aa9c2378b717dcb"}
Jul 25 01:17:47.755261 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:47.755221    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf" event={"ID":"4714b753-bcfd-4c76-91b5-650a94a6df3e","Type":"ContainerStarted","Data":"5d2c965002028b2f5a1eb2ba5218417ec258f6c9e520ff8976343b2136508b35"}
Jul 25 01:17:47.755705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:47.755311    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf"
Jul 25 01:17:47.757130 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:47.757112    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg" event={"ID":"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0","Type":"ContainerStarted","Data":"097295684de21a92d5d825071e7516f984bf57eb5b52fd0e950aecdbbb80e2d2"}
Jul 25 01:17:47.771745 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:47.771691    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf" podStartSLOduration=1.409031696 podStartE2EDuration="4.771670802s" totalImagesPullingTime="3.362639106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:17:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:17:47.770542715 +0000 UTC m=+351.880351999" watchObservedRunningTime="2026-07-25 01:17:47.771670802 +0000 UTC m=+351.881480087"
Jul 25 01:17:48.762286 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:48.762209    2572 generic.go:356] "Generic (PLEG): container finished" podID="6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" containerID="097295684de21a92d5d825071e7516f984bf57eb5b52fd0e950aecdbbb80e2d2" exitCode=0
Jul 25 01:17:48.762700 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:48.762301    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg" event={"ID":"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0","Type":"ContainerDied","Data":"097295684de21a92d5d825071e7516f984bf57eb5b52fd0e950aecdbbb80e2d2"}
Jul 25 01:17:49.767768 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:49.767736    2572 generic.go:356] "Generic (PLEG): container finished" podID="6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" containerID="d95a54d6529a9552d53b3e41e519633cb43953f2d00b68aaf236b8d596251adb" exitCode=0
Jul 25 01:17:49.768162 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:49.767806    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg" event={"ID":"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0","Type":"ContainerDied","Data":"d95a54d6529a9552d53b3e41e519633cb43953f2d00b68aaf236b8d596251adb"}
Jul 25 01:17:50.892156 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:50.892125    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:50.939615 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:50.939583    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-util\") pod \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") "
Jul 25 01:17:50.939758 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:50.939628    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-bundle\") pod \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") "
Jul 25 01:17:50.939758 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:50.939657    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77rw2\" (UniqueName: \"kubernetes.io/projected/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-kube-api-access-77rw2\") pod \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\" (UID: \"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0\") "
Jul 25 01:17:50.940053 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:50.940023    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-bundle" (OuterVolumeSpecName: "bundle") pod "6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" (UID: "6f4f9cc8-68d5-4f7b-8d85-2d04714713b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:17:50.941703 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:50.941674    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-kube-api-access-77rw2" (OuterVolumeSpecName: "kube-api-access-77rw2") pod "6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" (UID: "6f4f9cc8-68d5-4f7b-8d85-2d04714713b0"). InnerVolumeSpecName "kube-api-access-77rw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:17:50.944409 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:50.944385    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-util" (OuterVolumeSpecName: "util") pod "6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" (UID: "6f4f9cc8-68d5-4f7b-8d85-2d04714713b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:17:51.040646 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:51.040584    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-util\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:17:51.040646 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:51.040607    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-bundle\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:17:51.040646 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:51.040617    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77rw2\" (UniqueName: \"kubernetes.io/projected/6f4f9cc8-68d5-4f7b-8d85-2d04714713b0-kube-api-access-77rw2\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:17:51.777884 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:51.777857    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg"
Jul 25 01:17:51.778043 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:51.777855    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fk5xsg" event={"ID":"6f4f9cc8-68d5-4f7b-8d85-2d04714713b0","Type":"ContainerDied","Data":"126d8dcbfa8bb70169ffd0600bf3df6d7695a4e6d7f10a651aa9c2378b717dcb"}
Jul 25 01:17:51.778043 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:51.777979    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="126d8dcbfa8bb70169ffd0600bf3df6d7695a4e6d7f10a651aa9c2378b717dcb"
Jul 25 01:17:53.764247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:53.764217    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-5mqqf"
Jul 25 01:17:58.954387 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.954356    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh"]
Jul 25 01:17:58.954755 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.954613    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" containerName="util"
Jul 25 01:17:58.954755 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.954623    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" containerName="util"
Jul 25 01:17:58.954755 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.954631    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" containerName="extract"
Jul 25 01:17:58.954755 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.954637    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" containerName="extract"
Jul 25 01:17:58.954755 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.954646    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" containerName="pull"
Jul 25 01:17:58.954755 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.954651    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" containerName="pull"
Jul 25 01:17:58.954755 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.954701    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f4f9cc8-68d5-4f7b-8d85-2d04714713b0" containerName="extract"
Jul 25 01:17:58.958149 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.958132    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh"
Jul 25 01:17:58.960626 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.960604    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 25 01:17:58.961512 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.961493    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 25 01:17:58.961620 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.961543    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pk469\""
Jul 25 01:17:58.966561 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:58.966540    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh"]
Jul 25 01:17:59.001235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:59.001209    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmc54\" (UniqueName: \"kubernetes.io/projected/598247d6-3791-4ff8-a3e4-a15fdc72b323-kube-api-access-vmc54\") pod \"openshift-lws-operator-fd8ccff4c-9xxwh\" (UID: \"598247d6-3791-4ff8-a3e4-a15fdc72b323\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh"
Jul 25 01:17:59.001340 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:59.001243    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/598247d6-3791-4ff8-a3e4-a15fdc72b323-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9xxwh\" (UID: \"598247d6-3791-4ff8-a3e4-a15fdc72b323\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh"
Jul 25 01:17:59.102536 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:59.102512    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmc54\" (UniqueName: \"kubernetes.io/projected/598247d6-3791-4ff8-a3e4-a15fdc72b323-kube-api-access-vmc54\") pod \"openshift-lws-operator-fd8ccff4c-9xxwh\" (UID: \"598247d6-3791-4ff8-a3e4-a15fdc72b323\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh"
Jul 25 01:17:59.102638 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:59.102547    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/598247d6-3791-4ff8-a3e4-a15fdc72b323-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9xxwh\" (UID: \"598247d6-3791-4ff8-a3e4-a15fdc72b323\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh"
Jul 25 01:17:59.102918 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:59.102901    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/598247d6-3791-4ff8-a3e4-a15fdc72b323-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9xxwh\" (UID: \"598247d6-3791-4ff8-a3e4-a15fdc72b323\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh"
Jul 25 01:17:59.110631 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:59.110607    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmc54\" (UniqueName: \"kubernetes.io/projected/598247d6-3791-4ff8-a3e4-a15fdc72b323-kube-api-access-vmc54\") pod \"openshift-lws-operator-fd8ccff4c-9xxwh\" (UID: \"598247d6-3791-4ff8-a3e4-a15fdc72b323\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh"
Jul 25 01:17:59.267807 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:59.267726    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh"
Jul 25 01:17:59.407145 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:59.407119    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh"]
Jul 25 01:17:59.409756 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:17:59.409727    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod598247d6_3791_4ff8_a3e4_a15fdc72b323.slice/crio-409b015cfca40d78718b28037b2a4bc4be1205a8bef28b5d43e8aafd8f4d7d40 WatchSource:0}: Error finding container 409b015cfca40d78718b28037b2a4bc4be1205a8bef28b5d43e8aafd8f4d7d40: Status 404 returned error can't find the container with id 409b015cfca40d78718b28037b2a4bc4be1205a8bef28b5d43e8aafd8f4d7d40
Jul 25 01:17:59.806799 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:17:59.806762    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh" event={"ID":"598247d6-3791-4ff8-a3e4-a15fdc72b323","Type":"ContainerStarted","Data":"409b015cfca40d78718b28037b2a4bc4be1205a8bef28b5d43e8aafd8f4d7d40"}
Jul 25 01:18:01.817233 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:01.817154    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh" event={"ID":"598247d6-3791-4ff8-a3e4-a15fdc72b323","Type":"ContainerStarted","Data":"582aab6bf9eecb7a59582c33c44829f387d4dbc3df5cfafce40a79eb7a255bd3"}
Jul 25 01:18:01.832710 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:01.832659    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9xxwh" podStartSLOduration=1.741081152 podStartE2EDuration="3.832644636s" totalImagesPullingTime="2.091563484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:17:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:18:01.831008954 +0000 UTC m=+365.940818261" watchObservedRunningTime="2026-07-25 01:18:01.832644636 +0000 UTC m=+365.942453919"
Jul 25 01:18:04.117581 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.117540    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"]
Jul 25 01:18:04.136263 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.136236    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"]
Jul 25 01:18:04.136444 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.136382    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:04.138738 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.138713    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-88vfc\""
Jul 25 01:18:04.138738 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.138728    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 25 01:18:04.139624 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.139602    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 25 01:18:04.242384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.242348    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:04.242384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.242387    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk7b5\" (UniqueName: \"kubernetes.io/projected/260d7b72-d747-469c-b2f0-eb21cb5945fa-kube-api-access-vk7b5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:04.242583 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.242415    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:04.343228 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.343199    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:04.343447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.343236    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk7b5\" (UniqueName: \"kubernetes.io/projected/260d7b72-d747-469c-b2f0-eb21cb5945fa-kube-api-access-vk7b5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:04.343447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.343271    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:04.343597 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.343577    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:04.343665 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.343644    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:04.353601 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.353569    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk7b5\" (UniqueName: \"kubernetes.io/projected/260d7b72-d747-469c-b2f0-eb21cb5945fa-kube-api-access-vk7b5\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:04.445596 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.445567    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:04.598292 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.598269    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"]
Jul 25 01:18:04.600405 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:18:04.600378    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260d7b72_d747_469c_b2f0_eb21cb5945fa.slice/crio-f35ab681b57e85d743d86fe4f0bfd7c3affd8d5131dfd6c6b1625f579cd6ac88 WatchSource:0}: Error finding container f35ab681b57e85d743d86fe4f0bfd7c3affd8d5131dfd6c6b1625f579cd6ac88: Status 404 returned error can't find the container with id f35ab681b57e85d743d86fe4f0bfd7c3affd8d5131dfd6c6b1625f579cd6ac88
Jul 25 01:18:04.830001 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.829968    2572 generic.go:356] "Generic (PLEG): container finished" podID="260d7b72-d747-469c-b2f0-eb21cb5945fa" containerID="ac4203688c45a59ff4993d776abee90fa69ebac9e4ab66b1a81f09d3f0dce3ff" exitCode=0
Jul 25 01:18:04.830147 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.830054    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq" event={"ID":"260d7b72-d747-469c-b2f0-eb21cb5945fa","Type":"ContainerDied","Data":"ac4203688c45a59ff4993d776abee90fa69ebac9e4ab66b1a81f09d3f0dce3ff"}
Jul 25 01:18:04.830147 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:04.830089    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq" event={"ID":"260d7b72-d747-469c-b2f0-eb21cb5945fa","Type":"ContainerStarted","Data":"f35ab681b57e85d743d86fe4f0bfd7c3affd8d5131dfd6c6b1625f579cd6ac88"}
Jul 25 01:18:05.835376 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:05.835319    2572 generic.go:356] "Generic (PLEG): container finished" podID="260d7b72-d747-469c-b2f0-eb21cb5945fa" containerID="ad72b931d1a80946729aae4d56cdc738020ddbaa6d88af10c4bd1210b687400e" exitCode=0
Jul 25 01:18:05.835376 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:05.835359    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq" event={"ID":"260d7b72-d747-469c-b2f0-eb21cb5945fa","Type":"ContainerDied","Data":"ad72b931d1a80946729aae4d56cdc738020ddbaa6d88af10c4bd1210b687400e"}
Jul 25 01:18:06.840862 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:06.840824    2572 generic.go:356] "Generic (PLEG): container finished" podID="260d7b72-d747-469c-b2f0-eb21cb5945fa" containerID="c83c9dad236657c649d3853eafac71567af794f0a8eb8babf6798647661938f6" exitCode=0
Jul 25 01:18:06.841218 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:06.840899    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq" event={"ID":"260d7b72-d747-469c-b2f0-eb21cb5945fa","Type":"ContainerDied","Data":"c83c9dad236657c649d3853eafac71567af794f0a8eb8babf6798647661938f6"}
Jul 25 01:18:07.970647 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:07.970625    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:08.074868 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.074826    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-bundle\") pod \"260d7b72-d747-469c-b2f0-eb21cb5945fa\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") "
Jul 25 01:18:08.074868 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.074875    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-util\") pod \"260d7b72-d747-469c-b2f0-eb21cb5945fa\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") "
Jul 25 01:18:08.075094 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.074901    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk7b5\" (UniqueName: \"kubernetes.io/projected/260d7b72-d747-469c-b2f0-eb21cb5945fa-kube-api-access-vk7b5\") pod \"260d7b72-d747-469c-b2f0-eb21cb5945fa\" (UID: \"260d7b72-d747-469c-b2f0-eb21cb5945fa\") "
Jul 25 01:18:08.075576 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.075548    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-bundle" (OuterVolumeSpecName: "bundle") pod "260d7b72-d747-469c-b2f0-eb21cb5945fa" (UID: "260d7b72-d747-469c-b2f0-eb21cb5945fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:18:08.076915 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.076880    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260d7b72-d747-469c-b2f0-eb21cb5945fa-kube-api-access-vk7b5" (OuterVolumeSpecName: "kube-api-access-vk7b5") pod "260d7b72-d747-469c-b2f0-eb21cb5945fa" (UID: "260d7b72-d747-469c-b2f0-eb21cb5945fa"). InnerVolumeSpecName "kube-api-access-vk7b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:18:08.080309 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.080287    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-util" (OuterVolumeSpecName: "util") pod "260d7b72-d747-469c-b2f0-eb21cb5945fa" (UID: "260d7b72-d747-469c-b2f0-eb21cb5945fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:18:08.175886 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.175839    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-bundle\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:18:08.175886 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.175879    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/260d7b72-d747-469c-b2f0-eb21cb5945fa-util\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:18:08.175886 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.175889    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk7b5\" (UniqueName: \"kubernetes.io/projected/260d7b72-d747-469c-b2f0-eb21cb5945fa-kube-api-access-vk7b5\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:18:08.849264 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.849229    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq" event={"ID":"260d7b72-d747-469c-b2f0-eb21cb5945fa","Type":"ContainerDied","Data":"f35ab681b57e85d743d86fe4f0bfd7c3affd8d5131dfd6c6b1625f579cd6ac88"}
Jul 25 01:18:08.849264 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.849256    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afkjfzq"
Jul 25 01:18:08.849481 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:08.849276    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f35ab681b57e85d743d86fe4f0bfd7c3affd8d5131dfd6c6b1625f579cd6ac88"
Jul 25 01:18:16.757484 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.757447    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"]
Jul 25 01:18:16.757871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.757720    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260d7b72-d747-469c-b2f0-eb21cb5945fa" containerName="extract"
Jul 25 01:18:16.757871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.757731    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="260d7b72-d747-469c-b2f0-eb21cb5945fa" containerName="extract"
Jul 25 01:18:16.757871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.757743    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260d7b72-d747-469c-b2f0-eb21cb5945fa" containerName="pull"
Jul 25 01:18:16.757871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.757748    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="260d7b72-d747-469c-b2f0-eb21cb5945fa" containerName="pull"
Jul 25 01:18:16.757871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.757761    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260d7b72-d747-469c-b2f0-eb21cb5945fa" containerName="util"
Jul 25 01:18:16.757871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.757766    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="260d7b72-d747-469c-b2f0-eb21cb5945fa" containerName="util"
Jul 25 01:18:16.757871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.757812    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="260d7b72-d747-469c-b2f0-eb21cb5945fa" containerName="extract"
Jul 25 01:18:16.759870 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.759853    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.764168 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.764142    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cdnkn\""
Jul 25 01:18:16.764499 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.764470    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 25 01:18:16.764499 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.764486    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 25 01:18:16.764610 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.764535    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 25 01:18:16.770968 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.770951    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"]
Jul 25 01:18:16.840271 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.840240    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5mst\" (UniqueName: \"kubernetes.io/projected/cdae8040-882c-4084-bee8-b04b30990873-kube-api-access-p5mst\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.840425 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.840297    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdae8040-882c-4084-bee8-b04b30990873-cert\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.840425 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.840383    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cdae8040-882c-4084-bee8-b04b30990873-manager-config\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.840542 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.840440    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdae8040-882c-4084-bee8-b04b30990873-metrics-cert\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.941434 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.941409    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdae8040-882c-4084-bee8-b04b30990873-cert\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.941577 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.941453    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cdae8040-882c-4084-bee8-b04b30990873-manager-config\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.941577 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.941504    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdae8040-882c-4084-bee8-b04b30990873-metrics-cert\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.941577 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.941556    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5mst\" (UniqueName: \"kubernetes.io/projected/cdae8040-882c-4084-bee8-b04b30990873-kube-api-access-p5mst\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.942126 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.942105    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cdae8040-882c-4084-bee8-b04b30990873-manager-config\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.943977 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.943951    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cdae8040-882c-4084-bee8-b04b30990873-cert\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.944065 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.943980    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdae8040-882c-4084-bee8-b04b30990873-metrics-cert\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:16.949849 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:16.949830    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5mst\" (UniqueName: \"kubernetes.io/projected/cdae8040-882c-4084-bee8-b04b30990873-kube-api-access-p5mst\") pod \"lws-controller-manager-6b87879b5d-k6bs4\" (UID: \"cdae8040-882c-4084-bee8-b04b30990873\") " pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:17.068438 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.068361    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:17.200843 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.200686    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"]
Jul 25 01:18:17.203777 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:18:17.203747    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdae8040_882c_4084_bee8_b04b30990873.slice/crio-41d8aff43f3abca33855ec0048c62c4f3c93b33da0aa9f18f427d795d6955620 WatchSource:0}: Error finding container 41d8aff43f3abca33855ec0048c62c4f3c93b33da0aa9f18f427d795d6955620: Status 404 returned error can't find the container with id 41d8aff43f3abca33855ec0048c62c4f3c93b33da0aa9f18f427d795d6955620
Jul 25 01:18:17.402883 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.400126    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"]
Jul 25 01:18:17.404270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.404247    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.412786 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.412760    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 25 01:18:17.412901 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.412809    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ndvlx\""
Jul 25 01:18:17.413070 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.413020    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 25 01:18:17.413174 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.413150    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 25 01:18:17.413265 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.413247    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 25 01:18:17.423686 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.423666    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"]
Jul 25 01:18:17.547799 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.547761    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28g44\" (UniqueName: \"kubernetes.io/projected/755b8adb-c446-4f61-b3b4-18e9b080e521-kube-api-access-28g44\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.547991 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.547816    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/755b8adb-c446-4f61-b3b4-18e9b080e521-manifests\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.547991 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.547836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/755b8adb-c446-4f61-b3b4-18e9b080e521-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.547991 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.547926    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/755b8adb-c446-4f61-b3b4-18e9b080e521-tmp\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.548139 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.548020    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/755b8adb-c446-4f61-b3b4-18e9b080e521-webhook-cert\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.649453 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.649416    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28g44\" (UniqueName: \"kubernetes.io/projected/755b8adb-c446-4f61-b3b4-18e9b080e521-kube-api-access-28g44\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.649658 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.649466    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/755b8adb-c446-4f61-b3b4-18e9b080e521-manifests\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.649658 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.649489    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/755b8adb-c446-4f61-b3b4-18e9b080e521-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.649658 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.649507    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/755b8adb-c446-4f61-b3b4-18e9b080e521-tmp\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.649658 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.649541    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/755b8adb-c446-4f61-b3b4-18e9b080e521-webhook-cert\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.649944 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.649923    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/755b8adb-c446-4f61-b3b4-18e9b080e521-manifests\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.651827 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.651799    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/755b8adb-c446-4f61-b3b4-18e9b080e521-tmp\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.651939 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.651922    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/755b8adb-c446-4f61-b3b4-18e9b080e521-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.652054 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.652018    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/755b8adb-c446-4f61-b3b4-18e9b080e521-webhook-cert\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.660152 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.660092    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28g44\" (UniqueName: \"kubernetes.io/projected/755b8adb-c446-4f61-b3b4-18e9b080e521-kube-api-access-28g44\") pod \"opendatahub-operator-controller-manager-7587ff6c45-nnbmn\" (UID: \"755b8adb-c446-4f61-b3b4-18e9b080e521\") " pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.716285 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.716256    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:17.854629 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.854604    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"]
Jul 25 01:18:17.856334 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:18:17.856291    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755b8adb_c446_4f61_b3b4_18e9b080e521.slice/crio-4c5f146aefb9c3e935b7e978b16ad6c69c2e502facc2f26d67664c18397e956e WatchSource:0}: Error finding container 4c5f146aefb9c3e935b7e978b16ad6c69c2e502facc2f26d67664c18397e956e: Status 404 returned error can't find the container with id 4c5f146aefb9c3e935b7e978b16ad6c69c2e502facc2f26d67664c18397e956e
Jul 25 01:18:17.885050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.885020    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4" event={"ID":"cdae8040-882c-4084-bee8-b04b30990873","Type":"ContainerStarted","Data":"41d8aff43f3abca33855ec0048c62c4f3c93b33da0aa9f18f427d795d6955620"}
Jul 25 01:18:17.886212 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:17.886182    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn" event={"ID":"755b8adb-c446-4f61-b3b4-18e9b080e521","Type":"ContainerStarted","Data":"4c5f146aefb9c3e935b7e978b16ad6c69c2e502facc2f26d67664c18397e956e"}
Jul 25 01:18:19.898415 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:19.898349    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4" event={"ID":"cdae8040-882c-4084-bee8-b04b30990873","Type":"ContainerStarted","Data":"854fbe6b3b6ba2a46adc47015a427af5fbe048d4117e5002db09af786fb1604e"}
Jul 25 01:18:19.898845 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:19.898496    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:19.921462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:19.921415    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4" podStartSLOduration=1.9856025929999999 podStartE2EDuration="3.921399294s" totalImagesPullingTime="1.935796701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:18:19.919946865 +0000 UTC m=+384.029756150" watchObservedRunningTime="2026-07-25 01:18:19.921399294 +0000 UTC m=+384.031208577"
Jul 25 01:18:21.907831 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:21.907795    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn" event={"ID":"755b8adb-c446-4f61-b3b4-18e9b080e521","Type":"ContainerStarted","Data":"771dd3da15aa2d0604851ec3bd79a88fc5a11dad979997cfc16116c315e0a740"}
Jul 25 01:18:22.913165 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:22.913133    2572 generic.go:356] "Generic (PLEG): container finished" podID="755b8adb-c446-4f61-b3b4-18e9b080e521" containerID="771dd3da15aa2d0604851ec3bd79a88fc5a11dad979997cfc16116c315e0a740" exitCode=0
Jul 25 01:18:22.913559 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:22.913192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn" event={"ID":"755b8adb-c446-4f61-b3b4-18e9b080e521","Type":"ContainerDied","Data":"771dd3da15aa2d0604851ec3bd79a88fc5a11dad979997cfc16116c315e0a740"}
Jul 25 01:18:23.919615 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:23.919575    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn" event={"ID":"755b8adb-c446-4f61-b3b4-18e9b080e521","Type":"ContainerStarted","Data":"74159d0094d8261cd1063c19505f66ee93253c5d148c2090ecaedba0b4588f54"}
Jul 25 01:18:23.919985 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:23.919725    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:23.945711 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:23.945660    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn" podStartSLOduration=2.50525011 podStartE2EDuration="6.945646219s" totalImagesPullingTime="3.409300148s" totalInitContainerRuntime="1.031095961s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:18:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:18:23.938684716 +0000 UTC m=+388.048494013" watchObservedRunningTime="2026-07-25 01:18:23.945646219 +0000 UTC m=+388.055455503"
Jul 25 01:18:30.904174 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:30.904134    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6b87879b5d-k6bs4"
Jul 25 01:18:34.925893 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:34.925863    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7587ff6c45-nnbmn"
Jul 25 01:18:38.174396 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.174355    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-789f49c94d-6988f"]
Jul 25 01:18:38.182761 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.182732    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:38.187832 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.187805    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 25 01:18:38.189121 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.189098    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 25 01:18:38.189224 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.189148    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 25 01:18:38.190582 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.190082    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9hjgq\""
Jul 25 01:18:38.190582 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.190429    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 25 01:18:38.191159 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.191133    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-789f49c94d-6988f"]
Jul 25 01:18:38.214747 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.214721    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7w9v\" (UniqueName: \"kubernetes.io/projected/13f5b243-a80b-40d5-90f5-d920044e35e2-kube-api-access-w7w9v\") pod \"kube-auth-proxy-789f49c94d-6988f\" (UID: \"13f5b243-a80b-40d5-90f5-d920044e35e2\") " pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:38.214858 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.214770    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13f5b243-a80b-40d5-90f5-d920044e35e2-tmp\") pod \"kube-auth-proxy-789f49c94d-6988f\" (UID: \"13f5b243-a80b-40d5-90f5-d920044e35e2\") " pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:38.214858 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.214840    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f5b243-a80b-40d5-90f5-d920044e35e2-tls-certs\") pod \"kube-auth-proxy-789f49c94d-6988f\" (UID: \"13f5b243-a80b-40d5-90f5-d920044e35e2\") " pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:38.315259 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.315224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7w9v\" (UniqueName: \"kubernetes.io/projected/13f5b243-a80b-40d5-90f5-d920044e35e2-kube-api-access-w7w9v\") pod \"kube-auth-proxy-789f49c94d-6988f\" (UID: \"13f5b243-a80b-40d5-90f5-d920044e35e2\") " pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:38.315445 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.315294    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13f5b243-a80b-40d5-90f5-d920044e35e2-tmp\") pod \"kube-auth-proxy-789f49c94d-6988f\" (UID: \"13f5b243-a80b-40d5-90f5-d920044e35e2\") " pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:38.315445 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.315356    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f5b243-a80b-40d5-90f5-d920044e35e2-tls-certs\") pod \"kube-auth-proxy-789f49c94d-6988f\" (UID: \"13f5b243-a80b-40d5-90f5-d920044e35e2\") " pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:38.315536 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:18:38.315470    2572 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 25 01:18:38.315575 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:18:38.315537    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13f5b243-a80b-40d5-90f5-d920044e35e2-tls-certs podName:13f5b243-a80b-40d5-90f5-d920044e35e2 nodeName:}" failed. No retries permitted until 2026-07-25 01:18:38.815516533 +0000 UTC m=+402.925325797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/13f5b243-a80b-40d5-90f5-d920044e35e2-tls-certs") pod "kube-auth-proxy-789f49c94d-6988f" (UID: "13f5b243-a80b-40d5-90f5-d920044e35e2") : secret "kube-auth-proxy-tls" not found
Jul 25 01:18:38.317851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.317821    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13f5b243-a80b-40d5-90f5-d920044e35e2-tmp\") pod \"kube-auth-proxy-789f49c94d-6988f\" (UID: \"13f5b243-a80b-40d5-90f5-d920044e35e2\") " pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:38.323740 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.323716    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7w9v\" (UniqueName: \"kubernetes.io/projected/13f5b243-a80b-40d5-90f5-d920044e35e2-kube-api-access-w7w9v\") pod \"kube-auth-proxy-789f49c94d-6988f\" (UID: \"13f5b243-a80b-40d5-90f5-d920044e35e2\") " pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:38.818606 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.818564    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f5b243-a80b-40d5-90f5-d920044e35e2-tls-certs\") pod \"kube-auth-proxy-789f49c94d-6988f\" (UID: \"13f5b243-a80b-40d5-90f5-d920044e35e2\") " pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:38.821025 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:38.820997    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13f5b243-a80b-40d5-90f5-d920044e35e2-tls-certs\") pod \"kube-auth-proxy-789f49c94d-6988f\" (UID: \"13f5b243-a80b-40d5-90f5-d920044e35e2\") " pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:39.094552 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:39.094477    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f"
Jul 25 01:18:39.235632 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:39.235594    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-789f49c94d-6988f"]
Jul 25 01:18:39.238102 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:18:39.238078    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f5b243_a80b_40d5_90f5_d920044e35e2.slice/crio-d122c6dc45eac10a3e9e33bb8343197ec6cb717ee95ef9877ec84039a2e7cc99 WatchSource:0}: Error finding container d122c6dc45eac10a3e9e33bb8343197ec6cb717ee95ef9877ec84039a2e7cc99: Status 404 returned error can't find the container with id d122c6dc45eac10a3e9e33bb8343197ec6cb717ee95ef9877ec84039a2e7cc99
Jul 25 01:18:39.989712 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:39.989678    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f" event={"ID":"13f5b243-a80b-40d5-90f5-d920044e35e2","Type":"ContainerStarted","Data":"d122c6dc45eac10a3e9e33bb8343197ec6cb717ee95ef9877ec84039a2e7cc99"}
Jul 25 01:18:41.999278 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:41.999198    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f" event={"ID":"13f5b243-a80b-40d5-90f5-d920044e35e2","Type":"ContainerStarted","Data":"5896b17a5d746f297bf69f81bb1878f05b365964fea3356490727981d4582590"}
Jul 25 01:18:42.016963 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:18:42.016843    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-789f49c94d-6988f" podStartSLOduration=1.521778224 podStartE2EDuration="4.016823786s" totalImagesPullingTime="2.495045562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:18:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:18:42.016808281 +0000 UTC m=+406.126617585" watchObservedRunningTime="2026-07-25 01:18:42.016823786 +0000 UTC m=+406.126633072"
Jul 25 01:19:54.294801 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.294762    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"]
Jul 25 01:19:54.298432 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.298409    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:54.300614 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.300596    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 25 01:19:54.301402 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.301386    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zjr6l\""
Jul 25 01:19:54.301487 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.301425    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 25 01:19:54.306483 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.306456    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"]
Jul 25 01:19:54.310605 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.310580    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:54.310697 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.310630    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktzdx\" (UniqueName: \"kubernetes.io/projected/ab48424a-6670-4bbc-885e-843199410862-kube-api-access-ktzdx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:54.310753 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.310735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:54.411731 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.411702    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:54.411888 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.411751    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:54.411888 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.411874    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktzdx\" (UniqueName: \"kubernetes.io/projected/ab48424a-6670-4bbc-885e-843199410862-kube-api-access-ktzdx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:54.412055 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.412040    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:54.412093 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.412068    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:54.419683 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.419658    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktzdx\" (UniqueName: \"kubernetes.io/projected/ab48424a-6670-4bbc-885e-843199410862-kube-api-access-ktzdx\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:54.607724 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.607650    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:54.735870 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:54.735804    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"]
Jul 25 01:19:54.738099 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:19:54.738073    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab48424a_6670_4bbc_885e_843199410862.slice/crio-4ff991cebc35e6931b2e0616f5be609e32f1366f5e1cbd0bd1812dcc083c9423 WatchSource:0}: Error finding container 4ff991cebc35e6931b2e0616f5be609e32f1366f5e1cbd0bd1812dcc083c9423: Status 404 returned error can't find the container with id 4ff991cebc35e6931b2e0616f5be609e32f1366f5e1cbd0bd1812dcc083c9423
Jul 25 01:19:55.096239 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.096203    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"]
Jul 25 01:19:55.099693 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.099677    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:55.108162 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.108126    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"]
Jul 25 01:19:55.118028 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.117899    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:55.118028 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.117990    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97vh4\" (UniqueName: \"kubernetes.io/projected/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-kube-api-access-97vh4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:55.118207 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.118063    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:55.219185 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.219133    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97vh4\" (UniqueName: \"kubernetes.io/projected/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-kube-api-access-97vh4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:55.219185 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.219195    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:55.219454 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.219230    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:55.219637 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.219616    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:55.219637 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.219630    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:55.226909 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.226887    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97vh4\" (UniqueName: \"kubernetes.io/projected/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-kube-api-access-97vh4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:55.314186 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.314145    2572 generic.go:356] "Generic (PLEG): container finished" podID="ab48424a-6670-4bbc-885e-843199410862" containerID="0e47bc5b6f9e4993db4102db0a1fe9435a9e6ab860e41e7c7e9ae7b079ae463c" exitCode=0
Jul 25 01:19:55.314573 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.314219    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h" event={"ID":"ab48424a-6670-4bbc-885e-843199410862","Type":"ContainerDied","Data":"0e47bc5b6f9e4993db4102db0a1fe9435a9e6ab860e41e7c7e9ae7b079ae463c"}
Jul 25 01:19:55.314573 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.314268    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h" event={"ID":"ab48424a-6670-4bbc-885e-843199410862","Type":"ContainerStarted","Data":"4ff991cebc35e6931b2e0616f5be609e32f1366f5e1cbd0bd1812dcc083c9423"}
Jul 25 01:19:55.409938 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.409913    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:55.538363 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.537188    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"]
Jul 25 01:19:55.704388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.704287    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"]
Jul 25 01:19:55.707749 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.707728    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:55.715684 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.715659    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"]
Jul 25 01:19:55.723077 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.723051    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:55.723193 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.723098    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd4w4\" (UniqueName: \"kubernetes.io/projected/05aba405-3737-42e4-9d71-d24fd75f885b-kube-api-access-pd4w4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:55.723266 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.723197    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:55.824017 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.823982    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4w4\" (UniqueName: \"kubernetes.io/projected/05aba405-3737-42e4-9d71-d24fd75f885b-kube-api-access-pd4w4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:55.824189 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.824082    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:55.824189 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.824144    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:55.824542 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.824518    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:55.824634 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.824588    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:55.833451 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:55.833418    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd4w4\" (UniqueName: \"kubernetes.io/projected/05aba405-3737-42e4-9d71-d24fd75f885b-kube-api-access-pd4w4\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:56.017709 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.017687    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:56.096388 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.096356    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"]
Jul 25 01:19:56.099798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.099781    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:19:56.111284 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.111256    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"]
Jul 25 01:19:56.127160 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.127133    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:19:56.127315 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.127234    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:19:56.127315 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.127260    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5frz\" (UniqueName: \"kubernetes.io/projected/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-kube-api-access-z5frz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:19:56.150518 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.150496    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"]
Jul 25 01:19:56.152179 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:19:56.152154    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05aba405_3737_42e4_9d71_d24fd75f885b.slice/crio-779510f8e5ff7777ef25166fd36c87f009e2f5965819342e53b0de9c758b86b4 WatchSource:0}: Error finding container 779510f8e5ff7777ef25166fd36c87f009e2f5965819342e53b0de9c758b86b4: Status 404 returned error can't find the container with id 779510f8e5ff7777ef25166fd36c87f009e2f5965819342e53b0de9c758b86b4
Jul 25 01:19:56.228148 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.228112    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:19:56.228290 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.228154    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5frz\" (UniqueName: \"kubernetes.io/projected/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-kube-api-access-z5frz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:19:56.228290 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.228184    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:19:56.228568 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.228549    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:19:56.228617 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.228603    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:19:56.236012 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.235988    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5frz\" (UniqueName: \"kubernetes.io/projected/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-kube-api-access-z5frz\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:19:56.319639 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.319612    2572 generic.go:356] "Generic (PLEG): container finished" podID="402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" containerID="3dca23ec71f80b022c9a3480a9445dee08fa6c5fcedc287cc58fa135731d64a0" exitCode=0
Jul 25 01:19:56.320041 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.319692    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm" event={"ID":"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0","Type":"ContainerDied","Data":"3dca23ec71f80b022c9a3480a9445dee08fa6c5fcedc287cc58fa135731d64a0"}
Jul 25 01:19:56.320041 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.319741    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm" event={"ID":"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0","Type":"ContainerStarted","Data":"51f1d4577d4dd152e9454b22a76915d82030c2215bd56a68b7d8a7c14a37f3f3"}
Jul 25 01:19:56.321367 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.321342    2572 generic.go:356] "Generic (PLEG): container finished" podID="05aba405-3737-42e4-9d71-d24fd75f885b" containerID="fbb54ac5456588e3033290f48509a18ee2423be846882d5d67095f15e44e6e36" exitCode=0
Jul 25 01:19:56.321367 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.321370    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582" event={"ID":"05aba405-3737-42e4-9d71-d24fd75f885b","Type":"ContainerDied","Data":"fbb54ac5456588e3033290f48509a18ee2423be846882d5d67095f15e44e6e36"}
Jul 25 01:19:56.321583 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.321419    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582" event={"ID":"05aba405-3737-42e4-9d71-d24fd75f885b","Type":"ContainerStarted","Data":"779510f8e5ff7777ef25166fd36c87f009e2f5965819342e53b0de9c758b86b4"}
Jul 25 01:19:56.323435 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.323414    2572 generic.go:356] "Generic (PLEG): container finished" podID="ab48424a-6670-4bbc-885e-843199410862" containerID="ec80b3abe59f0699619fd571c29644b60abc48e800d966d55f9dbee77e008a6a" exitCode=0
Jul 25 01:19:56.323541 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.323495    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h" event={"ID":"ab48424a-6670-4bbc-885e-843199410862","Type":"ContainerDied","Data":"ec80b3abe59f0699619fd571c29644b60abc48e800d966d55f9dbee77e008a6a"}
Jul 25 01:19:56.451192 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.451161    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:19:56.579072 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:56.579050    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"]
Jul 25 01:19:56.581122 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:19:56.581098    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cad4dfe_1404_4c88_9a78_cfaac6b38eb6.slice/crio-9be268490e87f9b9c595da28e35958d618cadcac6ced4260b45ad0fbf15a25f2 WatchSource:0}: Error finding container 9be268490e87f9b9c595da28e35958d618cadcac6ced4260b45ad0fbf15a25f2: Status 404 returned error can't find the container with id 9be268490e87f9b9c595da28e35958d618cadcac6ced4260b45ad0fbf15a25f2
Jul 25 01:19:57.329545 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:57.329512    2572 generic.go:356] "Generic (PLEG): container finished" podID="402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" containerID="7ead21fd938183caf061b6da83f60fd559e76c53b0828205d39b0f7224838925" exitCode=0
Jul 25 01:19:57.329889 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:57.329616    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm" event={"ID":"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0","Type":"ContainerDied","Data":"7ead21fd938183caf061b6da83f60fd559e76c53b0828205d39b0f7224838925"}
Jul 25 01:19:57.331225 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:57.331202    2572 generic.go:356] "Generic (PLEG): container finished" podID="2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" containerID="d2381463f072e6ae0cd49fb18816a71baabab6c5ef511a6b1890722d645b97a6" exitCode=0
Jul 25 01:19:57.331287 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:57.331228    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm" event={"ID":"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6","Type":"ContainerDied","Data":"d2381463f072e6ae0cd49fb18816a71baabab6c5ef511a6b1890722d645b97a6"}
Jul 25 01:19:57.331287 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:57.331270    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm" event={"ID":"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6","Type":"ContainerStarted","Data":"9be268490e87f9b9c595da28e35958d618cadcac6ced4260b45ad0fbf15a25f2"}
Jul 25 01:19:57.333304 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:57.333201    2572 generic.go:356] "Generic (PLEG): container finished" podID="05aba405-3737-42e4-9d71-d24fd75f885b" containerID="c0e15c62b47f1317c3f9cae20b605b6bb07bcdf83414a333e3994097a9b2b9d9" exitCode=0
Jul 25 01:19:57.333304 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:57.333283    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582" event={"ID":"05aba405-3737-42e4-9d71-d24fd75f885b","Type":"ContainerDied","Data":"c0e15c62b47f1317c3f9cae20b605b6bb07bcdf83414a333e3994097a9b2b9d9"}
Jul 25 01:19:57.335601 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:57.335579    2572 generic.go:356] "Generic (PLEG): container finished" podID="ab48424a-6670-4bbc-885e-843199410862" containerID="e516b3f7f0d85502a9b19d2d4bdef50ed4a4e8a1ef53dccecb2456f684c47c77" exitCode=0
Jul 25 01:19:57.335682 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:57.335607    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h" event={"ID":"ab48424a-6670-4bbc-885e-843199410862","Type":"ContainerDied","Data":"e516b3f7f0d85502a9b19d2d4bdef50ed4a4e8a1ef53dccecb2456f684c47c77"}
Jul 25 01:19:57.463197 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:19:57.463166    2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05aba405_3737_42e4_9d71_d24fd75f885b.slice/crio-71a0d7a367a9e54c0ea8097cd9f6bba095b21ba8288da68e6152fb97425e5a62.scope\": RecentStats: unable to find data in memory cache]"
Jul 25 01:19:58.342128 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.342037    2572 generic.go:356] "Generic (PLEG): container finished" podID="2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" containerID="6cadf198e4614974fa434a45a5530488b9d63232faca73b561dac145d280e967" exitCode=0
Jul 25 01:19:58.342128 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.342095    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm" event={"ID":"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6","Type":"ContainerDied","Data":"6cadf198e4614974fa434a45a5530488b9d63232faca73b561dac145d280e967"}
Jul 25 01:19:58.344312 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.344254    2572 generic.go:356] "Generic (PLEG): container finished" podID="05aba405-3737-42e4-9d71-d24fd75f885b" containerID="71a0d7a367a9e54c0ea8097cd9f6bba095b21ba8288da68e6152fb97425e5a62" exitCode=0
Jul 25 01:19:58.344429 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.344358    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582" event={"ID":"05aba405-3737-42e4-9d71-d24fd75f885b","Type":"ContainerDied","Data":"71a0d7a367a9e54c0ea8097cd9f6bba095b21ba8288da68e6152fb97425e5a62"}
Jul 25 01:19:58.346494 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.346471    2572 generic.go:356] "Generic (PLEG): container finished" podID="402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" containerID="0b9f1c15774ce2090769f4e5ffd8f9add4842b27f125db98c96347e1c970f83a" exitCode=0
Jul 25 01:19:58.346598 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.346553    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm" event={"ID":"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0","Type":"ContainerDied","Data":"0b9f1c15774ce2090769f4e5ffd8f9add4842b27f125db98c96347e1c970f83a"}
Jul 25 01:19:58.485921 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.485896    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:58.547619 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.547589    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-util\") pod \"ab48424a-6670-4bbc-885e-843199410862\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") "
Jul 25 01:19:58.547758 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.547634    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktzdx\" (UniqueName: \"kubernetes.io/projected/ab48424a-6670-4bbc-885e-843199410862-kube-api-access-ktzdx\") pod \"ab48424a-6670-4bbc-885e-843199410862\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") "
Jul 25 01:19:58.547758 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.547676    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-bundle\") pod \"ab48424a-6670-4bbc-885e-843199410862\" (UID: \"ab48424a-6670-4bbc-885e-843199410862\") "
Jul 25 01:19:58.548310 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.548283    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-bundle" (OuterVolumeSpecName: "bundle") pod "ab48424a-6670-4bbc-885e-843199410862" (UID: "ab48424a-6670-4bbc-885e-843199410862"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:19:58.549813 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.549788    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab48424a-6670-4bbc-885e-843199410862-kube-api-access-ktzdx" (OuterVolumeSpecName: "kube-api-access-ktzdx") pod "ab48424a-6670-4bbc-885e-843199410862" (UID: "ab48424a-6670-4bbc-885e-843199410862"). InnerVolumeSpecName "kube-api-access-ktzdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:19:58.553534 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.553515    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-util" (OuterVolumeSpecName: "util") pod "ab48424a-6670-4bbc-885e-843199410862" (UID: "ab48424a-6670-4bbc-885e-843199410862"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:19:58.649028 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.648996    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-util\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:19:58.649028 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.649024    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktzdx\" (UniqueName: \"kubernetes.io/projected/ab48424a-6670-4bbc-885e-843199410862-kube-api-access-ktzdx\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:19:58.649196 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:58.649035    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab48424a-6670-4bbc-885e-843199410862-bundle\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:19:59.353623 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.353593    2572 generic.go:356] "Generic (PLEG): container finished" podID="2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" containerID="681a7c26d39053f745dac55b6fc0dad5487a74bef8eb77ee4155837eaf0e5f65" exitCode=0
Jul 25 01:19:59.354013 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.353670    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm" event={"ID":"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6","Type":"ContainerDied","Data":"681a7c26d39053f745dac55b6fc0dad5487a74bef8eb77ee4155837eaf0e5f65"}
Jul 25 01:19:59.355464 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.355440    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h" event={"ID":"ab48424a-6670-4bbc-885e-843199410862","Type":"ContainerDied","Data":"4ff991cebc35e6931b2e0616f5be609e32f1366f5e1cbd0bd1812dcc083c9423"}
Jul 25 01:19:59.355593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.355485    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ff991cebc35e6931b2e0616f5be609e32f1366f5e1cbd0bd1812dcc083c9423"
Jul 25 01:19:59.355593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.355448    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h"
Jul 25 01:19:59.512439 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.512419    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:19:59.514457 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.514440    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:19:59.555772 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.555748    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-util\") pod \"05aba405-3737-42e4-9d71-d24fd75f885b\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") "
Jul 25 01:19:59.555924 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.555798    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd4w4\" (UniqueName: \"kubernetes.io/projected/05aba405-3737-42e4-9d71-d24fd75f885b-kube-api-access-pd4w4\") pod \"05aba405-3737-42e4-9d71-d24fd75f885b\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") "
Jul 25 01:19:59.555924 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.555835    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-util\") pod \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") "
Jul 25 01:19:59.555924 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.555869    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97vh4\" (UniqueName: \"kubernetes.io/projected/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-kube-api-access-97vh4\") pod \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") "
Jul 25 01:19:59.555924 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.555916    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-bundle\") pod \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\" (UID: \"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0\") "
Jul 25 01:19:59.556145 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.555938    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-bundle\") pod \"05aba405-3737-42e4-9d71-d24fd75f885b\" (UID: \"05aba405-3737-42e4-9d71-d24fd75f885b\") "
Jul 25 01:19:59.556568 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.556528    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-bundle" (OuterVolumeSpecName: "bundle") pod "402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" (UID: "402756f6-2e2e-46e9-bd2e-2f9f0ed469d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:19:59.556694 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.556650    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-bundle" (OuterVolumeSpecName: "bundle") pod "05aba405-3737-42e4-9d71-d24fd75f885b" (UID: "05aba405-3737-42e4-9d71-d24fd75f885b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:19:59.558169 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.558143    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-kube-api-access-97vh4" (OuterVolumeSpecName: "kube-api-access-97vh4") pod "402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" (UID: "402756f6-2e2e-46e9-bd2e-2f9f0ed469d0"). InnerVolumeSpecName "kube-api-access-97vh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:19:59.558315 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.558298    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05aba405-3737-42e4-9d71-d24fd75f885b-kube-api-access-pd4w4" (OuterVolumeSpecName: "kube-api-access-pd4w4") pod "05aba405-3737-42e4-9d71-d24fd75f885b" (UID: "05aba405-3737-42e4-9d71-d24fd75f885b"). InnerVolumeSpecName "kube-api-access-pd4w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:19:59.561413 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.561391    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-util" (OuterVolumeSpecName: "util") pod "05aba405-3737-42e4-9d71-d24fd75f885b" (UID: "05aba405-3737-42e4-9d71-d24fd75f885b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:19:59.561941 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.561921    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-util" (OuterVolumeSpecName: "util") pod "402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" (UID: "402756f6-2e2e-46e9-bd2e-2f9f0ed469d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:19:59.657494 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.657458    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd4w4\" (UniqueName: \"kubernetes.io/projected/05aba405-3737-42e4-9d71-d24fd75f885b-kube-api-access-pd4w4\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:19:59.657494 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.657495    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-util\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:19:59.657494 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.657505    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97vh4\" (UniqueName: \"kubernetes.io/projected/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-kube-api-access-97vh4\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:19:59.657686 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.657514    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/402756f6-2e2e-46e9-bd2e-2f9f0ed469d0-bundle\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:19:59.657686 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.657523    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-bundle\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:19:59.657686 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:19:59.657529    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05aba405-3737-42e4-9d71-d24fd75f885b-util\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:20:00.362143 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.362120    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm"
Jul 25 01:20:00.362514 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.362152    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm" event={"ID":"402756f6-2e2e-46e9-bd2e-2f9f0ed469d0","Type":"ContainerDied","Data":"51f1d4577d4dd152e9454b22a76915d82030c2215bd56a68b7d8a7c14a37f3f3"}
Jul 25 01:20:00.362514 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.362200    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51f1d4577d4dd152e9454b22a76915d82030c2215bd56a68b7d8a7c14a37f3f3"
Jul 25 01:20:00.364158 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.364140    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582"
Jul 25 01:20:00.364240 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.364168    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582" event={"ID":"05aba405-3737-42e4-9d71-d24fd75f885b","Type":"ContainerDied","Data":"779510f8e5ff7777ef25166fd36c87f009e2f5965819342e53b0de9c758b86b4"}
Jul 25 01:20:00.364240 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.364201    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="779510f8e5ff7777ef25166fd36c87f009e2f5965819342e53b0de9c758b86b4"
Jul 25 01:20:00.482478 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.482452    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:20:00.564037 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.564002    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5frz\" (UniqueName: \"kubernetes.io/projected/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-kube-api-access-z5frz\") pod \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") "
Jul 25 01:20:00.564235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.564071    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-util\") pod \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") "
Jul 25 01:20:00.564235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.564107    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-bundle\") pod \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\" (UID: \"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6\") "
Jul 25 01:20:00.564749 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.564703    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-bundle" (OuterVolumeSpecName: "bundle") pod "2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" (UID: "2cad4dfe-1404-4c88-9a78-cfaac6b38eb6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:20:00.566137 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.566112    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-kube-api-access-z5frz" (OuterVolumeSpecName: "kube-api-access-z5frz") pod "2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" (UID: "2cad4dfe-1404-4c88-9a78-cfaac6b38eb6"). InnerVolumeSpecName "kube-api-access-z5frz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:20:00.569314 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.569273    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-util" (OuterVolumeSpecName: "util") pod "2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" (UID: "2cad4dfe-1404-4c88-9a78-cfaac6b38eb6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:20:00.665728 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.665696    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5frz\" (UniqueName: \"kubernetes.io/projected/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-kube-api-access-z5frz\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:20:00.665728 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.665726    2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-util\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:20:00.665871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:00.665737    2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2cad4dfe-1404-4c88-9a78-cfaac6b38eb6-bundle\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:20:01.371295 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:01.371268    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm"
Jul 25 01:20:01.371792 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:01.371268    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm" event={"ID":"2cad4dfe-1404-4c88-9a78-cfaac6b38eb6","Type":"ContainerDied","Data":"9be268490e87f9b9c595da28e35958d618cadcac6ced4260b45ad0fbf15a25f2"}
Jul 25 01:20:01.371792 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:01.371367    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9be268490e87f9b9c595da28e35958d618cadcac6ced4260b45ad0fbf15a25f2"
Jul 25 01:20:02.274810 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:02.274775    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76789559f6-p87vj"]
Jul 25 01:20:27.293533 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.293468    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76789559f6-p87vj" podUID="fac9b1d5-1cfa-48f2-8763-de945ac9ce34" containerName="console" containerID="cri-o://4335090e58429957bd30f3bb402df1c6b77e6c0d97d50a80680381134d2a7ef9" gracePeriod=15
Jul 25 01:20:27.487563 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.487531    2572 generic.go:356] "Generic (PLEG): container finished" podID="fac9b1d5-1cfa-48f2-8763-de945ac9ce34" containerID="4335090e58429957bd30f3bb402df1c6b77e6c0d97d50a80680381134d2a7ef9" exitCode=0
Jul 25 01:20:27.487711 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.487610    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76789559f6-p87vj" event={"ID":"fac9b1d5-1cfa-48f2-8763-de945ac9ce34","Type":"ContainerDied","Data":"4335090e58429957bd30f3bb402df1c6b77e6c0d97d50a80680381134d2a7ef9"}
Jul 25 01:20:27.542010 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.541982    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:20:27.675043 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675007    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-config\") pod \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") "
Jul 25 01:20:27.675231 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675057    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-oauth-serving-cert\") pod \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") "
Jul 25 01:20:27.675231 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675097    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-serving-cert\") pod \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") "
Jul 25 01:20:27.675231 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675151    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwpl6\" (UniqueName: \"kubernetes.io/projected/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-kube-api-access-cwpl6\") pod \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") "
Jul 25 01:20:27.675423 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675306    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-trusted-ca-bundle\") pod \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") "
Jul 25 01:20:27.675423 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675401    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-service-ca\") pod \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") "
Jul 25 01:20:27.675531 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675448    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-oauth-config\") pod \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\" (UID: \"fac9b1d5-1cfa-48f2-8763-de945ac9ce34\") "
Jul 25 01:20:27.675531 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675466    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-config" (OuterVolumeSpecName: "console-config") pod "fac9b1d5-1cfa-48f2-8763-de945ac9ce34" (UID: "fac9b1d5-1cfa-48f2-8763-de945ac9ce34"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:20:27.675640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675582    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fac9b1d5-1cfa-48f2-8763-de945ac9ce34" (UID: "fac9b1d5-1cfa-48f2-8763-de945ac9ce34"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:20:27.675704 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675678    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fac9b1d5-1cfa-48f2-8763-de945ac9ce34" (UID: "fac9b1d5-1cfa-48f2-8763-de945ac9ce34"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:20:27.675808 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675711    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-oauth-serving-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:20:27.675808 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675730    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-config\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:20:27.675808 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.675724    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-service-ca" (OuterVolumeSpecName: "service-ca") pod "fac9b1d5-1cfa-48f2-8763-de945ac9ce34" (UID: "fac9b1d5-1cfa-48f2-8763-de945ac9ce34"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:20:27.677305 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.677284    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fac9b1d5-1cfa-48f2-8763-de945ac9ce34" (UID: "fac9b1d5-1cfa-48f2-8763-de945ac9ce34"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:20:27.677305 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.677295    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-kube-api-access-cwpl6" (OuterVolumeSpecName: "kube-api-access-cwpl6") pod "fac9b1d5-1cfa-48f2-8763-de945ac9ce34" (UID: "fac9b1d5-1cfa-48f2-8763-de945ac9ce34"). InnerVolumeSpecName "kube-api-access-cwpl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:20:27.677539 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.677519    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fac9b1d5-1cfa-48f2-8763-de945ac9ce34" (UID: "fac9b1d5-1cfa-48f2-8763-de945ac9ce34"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:20:27.776472 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.776440    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwpl6\" (UniqueName: \"kubernetes.io/projected/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-kube-api-access-cwpl6\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:20:27.776472 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.776467    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-trusted-ca-bundle\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:20:27.776472 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.776478    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:20:27.776689 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.776487    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-oauth-config\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:20:27.776689 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:27.776495    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fac9b1d5-1cfa-48f2-8763-de945ac9ce34-console-serving-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:20:28.493393 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:28.493352    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76789559f6-p87vj" event={"ID":"fac9b1d5-1cfa-48f2-8763-de945ac9ce34","Type":"ContainerDied","Data":"163115bcdcd3085fbc567accf36b2edf0d651558b84521e96235e70885da3248"}
Jul 25 01:20:28.493803 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:28.493369    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76789559f6-p87vj"
Jul 25 01:20:28.493803 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:28.493421    2572 scope.go:117] "RemoveContainer" containerID="4335090e58429957bd30f3bb402df1c6b77e6c0d97d50a80680381134d2a7ef9"
Jul 25 01:20:28.538607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:28.538578    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76789559f6-p87vj"]
Jul 25 01:20:28.555874 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:28.555850    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76789559f6-p87vj"]
Jul 25 01:20:29.536919 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.536884    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"]
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537160    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05aba405-3737-42e4-9d71-d24fd75f885b" containerName="pull"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537170    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="05aba405-3737-42e4-9d71-d24fd75f885b" containerName="pull"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537181    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" containerName="pull"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537187    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" containerName="pull"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537193    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" containerName="extract"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537198    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" containerName="extract"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537206    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" containerName="util"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537211    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" containerName="util"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537219    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" containerName="extract"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537226    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" containerName="extract"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537232    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fac9b1d5-1cfa-48f2-8763-de945ac9ce34" containerName="console"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537237    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fac9b1d5-1cfa-48f2-8763-de945ac9ce34" containerName="console"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537245    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab48424a-6670-4bbc-885e-843199410862" containerName="extract"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537249    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab48424a-6670-4bbc-885e-843199410862" containerName="extract"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537259    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab48424a-6670-4bbc-885e-843199410862" containerName="pull"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537264    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab48424a-6670-4bbc-885e-843199410862" containerName="pull"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537270    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05aba405-3737-42e4-9d71-d24fd75f885b" containerName="extract"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537274    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="05aba405-3737-42e4-9d71-d24fd75f885b" containerName="extract"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537281    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab48424a-6670-4bbc-885e-843199410862" containerName="util"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537285    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab48424a-6670-4bbc-885e-843199410862" containerName="util"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537294    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05aba405-3737-42e4-9d71-d24fd75f885b" containerName="util"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537299    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="05aba405-3737-42e4-9d71-d24fd75f885b" containerName="util"
Jul 25 01:20:29.537291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537304    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" containerName="pull"
Jul 25 01:20:29.537971 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537309    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" containerName="pull"
Jul 25 01:20:29.537971 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537316    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" containerName="util"
Jul 25 01:20:29.537971 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537333    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" containerName="util"
Jul 25 01:20:29.537971 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537378    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab48424a-6670-4bbc-885e-843199410862" containerName="extract"
Jul 25 01:20:29.537971 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537385    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cad4dfe-1404-4c88-9a78-cfaac6b38eb6" containerName="extract"
Jul 25 01:20:29.537971 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537391    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="402756f6-2e2e-46e9-bd2e-2f9f0ed469d0" containerName="extract"
Jul 25 01:20:29.537971 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537398    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fac9b1d5-1cfa-48f2-8763-de945ac9ce34" containerName="console"
Jul 25 01:20:29.537971 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.537404    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="05aba405-3737-42e4-9d71-d24fd75f885b" containerName="extract"
Jul 25 01:20:29.541422 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.541402    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"
Jul 25 01:20:29.544925 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.544903    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 25 01:20:29.545501 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.545480    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 25 01:20:29.545659 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.545495    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zjr6l\""
Jul 25 01:20:29.545781 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.545511    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 25 01:20:29.545781 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.545517    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 25 01:20:29.560262 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.560237    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"]
Jul 25 01:20:29.693942 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.693872    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0336442f-1ed9-439f-835b-38f87c1657cb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hk4cj\" (UID: \"0336442f-1ed9-439f-835b-38f87c1657cb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"
Jul 25 01:20:29.693942 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.693920    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j7dc\" (UniqueName: \"kubernetes.io/projected/0336442f-1ed9-439f-835b-38f87c1657cb-kube-api-access-7j7dc\") pod \"kuadrant-console-plugin-66b5774985-hk4cj\" (UID: \"0336442f-1ed9-439f-835b-38f87c1657cb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"
Jul 25 01:20:29.694257 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.694009    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0336442f-1ed9-439f-835b-38f87c1657cb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hk4cj\" (UID: \"0336442f-1ed9-439f-835b-38f87c1657cb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"
Jul 25 01:20:29.795312 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.795219    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0336442f-1ed9-439f-835b-38f87c1657cb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hk4cj\" (UID: \"0336442f-1ed9-439f-835b-38f87c1657cb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"
Jul 25 01:20:29.795312 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.795271    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0336442f-1ed9-439f-835b-38f87c1657cb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hk4cj\" (UID: \"0336442f-1ed9-439f-835b-38f87c1657cb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"
Jul 25 01:20:29.795556 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.795436    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j7dc\" (UniqueName: \"kubernetes.io/projected/0336442f-1ed9-439f-835b-38f87c1657cb-kube-api-access-7j7dc\") pod \"kuadrant-console-plugin-66b5774985-hk4cj\" (UID: \"0336442f-1ed9-439f-835b-38f87c1657cb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"
Jul 25 01:20:29.795859 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.795839    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0336442f-1ed9-439f-835b-38f87c1657cb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hk4cj\" (UID: \"0336442f-1ed9-439f-835b-38f87c1657cb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"
Jul 25 01:20:29.797680 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.797657    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0336442f-1ed9-439f-835b-38f87c1657cb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hk4cj\" (UID: \"0336442f-1ed9-439f-835b-38f87c1657cb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"
Jul 25 01:20:29.802602 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.802581    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j7dc\" (UniqueName: \"kubernetes.io/projected/0336442f-1ed9-439f-835b-38f87c1657cb-kube-api-access-7j7dc\") pod \"kuadrant-console-plugin-66b5774985-hk4cj\" (UID: \"0336442f-1ed9-439f-835b-38f87c1657cb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"
Jul 25 01:20:29.850558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.850522    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"
Jul 25 01:20:29.978133 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:29.978109    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj"]
Jul 25 01:20:29.979820 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:20:29.979793    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0336442f_1ed9_439f_835b_38f87c1657cb.slice/crio-04a6ed68ca123e0c2274eb0b5919cdb44a78839218409d0ad9c9ee7a21381915 WatchSource:0}: Error finding container 04a6ed68ca123e0c2274eb0b5919cdb44a78839218409d0ad9c9ee7a21381915: Status 404 returned error can't find the container with id 04a6ed68ca123e0c2274eb0b5919cdb44a78839218409d0ad9c9ee7a21381915
Jul 25 01:20:30.353759 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:30.353730    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fac9b1d5-1cfa-48f2-8763-de945ac9ce34" path="/var/lib/kubelet/pods/fac9b1d5-1cfa-48f2-8763-de945ac9ce34/volumes"
Jul 25 01:20:30.505759 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:30.505722    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj" event={"ID":"0336442f-1ed9-439f-835b-38f87c1657cb","Type":"ContainerStarted","Data":"04a6ed68ca123e0c2274eb0b5919cdb44a78839218409d0ad9c9ee7a21381915"}
Jul 25 01:20:51.627539 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:51.627503    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj" event={"ID":"0336442f-1ed9-439f-835b-38f87c1657cb","Type":"ContainerStarted","Data":"d560dcf5765718bc251ae6dfe2e0d98eecaaf61e574ec843b24d53b46ed30bb5"}
Jul 25 01:20:51.641617 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:20:51.641563    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hk4cj" podStartSLOduration=1.170512294 podStartE2EDuration="22.641546481s" totalImagesPullingTime="21.471034187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:20:51.640927169 +0000 UTC m=+535.750736496" watchObservedRunningTime="2026-07-25 01:20:51.641546481 +0000 UTC m=+535.751355765"
Jul 25 01:21:22.157619 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.157584    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k45md"]
Jul 25 01:21:22.160315 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.160295    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:22.162739 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.162716    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 25 01:21:22.171175 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.171155    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k45md"]
Jul 25 01:21:22.218755 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.218725    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-config-file\") pod \"limitador-limitador-64c94fd64f-k45md\" (UID: \"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:22.218911 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.218861    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86scv\" (UniqueName: \"kubernetes.io/projected/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-kube-api-access-86scv\") pod \"limitador-limitador-64c94fd64f-k45md\" (UID: \"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:22.256203 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.256169    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k45md"]
Jul 25 01:21:22.319471 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.319437    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86scv\" (UniqueName: \"kubernetes.io/projected/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-kube-api-access-86scv\") pod \"limitador-limitador-64c94fd64f-k45md\" (UID: \"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:22.319471 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.319475    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-config-file\") pod \"limitador-limitador-64c94fd64f-k45md\" (UID: \"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:22.320086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.320066    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-config-file\") pod \"limitador-limitador-64c94fd64f-k45md\" (UID: \"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:22.327432 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.327403    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86scv\" (UniqueName: \"kubernetes.io/projected/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-kube-api-access-86scv\") pod \"limitador-limitador-64c94fd64f-k45md\" (UID: \"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:22.470652 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.470571    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:22.601060 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.601033    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k45md"]
Jul 25 01:21:22.603399 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:21:22.603369    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4cbca7_1a77_45bd_a493_fb8364b1d1f2.slice/crio-b2d504ddb0f751fa0d79b1ec30151341289ddb5cca70a608628e0a4053abcf83 WatchSource:0}: Error finding container b2d504ddb0f751fa0d79b1ec30151341289ddb5cca70a608628e0a4053abcf83: Status 404 returned error can't find the container with id b2d504ddb0f751fa0d79b1ec30151341289ddb5cca70a608628e0a4053abcf83
Jul 25 01:21:22.773948 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.773861    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md" event={"ID":"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2","Type":"ContainerStarted","Data":"b2d504ddb0f751fa0d79b1ec30151341289ddb5cca70a608628e0a4053abcf83"}
Jul 25 01:21:22.979765 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.979735    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-mbrt5"]
Jul 25 01:21:22.983682 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.983658    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mbrt5"
Jul 25 01:21:22.985764 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.985736    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dkvzb\""
Jul 25 01:21:22.991448 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:22.991417    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-mbrt5"]
Jul 25 01:21:23.127218 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.127144    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmwm8\" (UniqueName: \"kubernetes.io/projected/9d31e1bf-8738-4a68-a010-c50a01287fa3-kube-api-access-cmwm8\") pod \"authorino-58cd955675-mbrt5\" (UID: \"9d31e1bf-8738-4a68-a010-c50a01287fa3\") " pod="kuadrant-system/authorino-58cd955675-mbrt5"
Jul 25 01:21:23.154979 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.154953    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-n7s2d"]
Jul 25 01:21:23.158080 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.158059    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-n7s2d"
Jul 25 01:21:23.164982 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.164959    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-n7s2d"]
Jul 25 01:21:23.228199 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.228161    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmwm8\" (UniqueName: \"kubernetes.io/projected/9d31e1bf-8738-4a68-a010-c50a01287fa3-kube-api-access-cmwm8\") pod \"authorino-58cd955675-mbrt5\" (UID: \"9d31e1bf-8738-4a68-a010-c50a01287fa3\") " pod="kuadrant-system/authorino-58cd955675-mbrt5"
Jul 25 01:21:23.228398 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.228286    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrmrf\" (UniqueName: \"kubernetes.io/projected/a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952-kube-api-access-zrmrf\") pod \"authorino-75d556bd77-n7s2d\" (UID: \"a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952\") " pod="kuadrant-system/authorino-75d556bd77-n7s2d"
Jul 25 01:21:23.236793 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.236742    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmwm8\" (UniqueName: \"kubernetes.io/projected/9d31e1bf-8738-4a68-a010-c50a01287fa3-kube-api-access-cmwm8\") pod \"authorino-58cd955675-mbrt5\" (UID: \"9d31e1bf-8738-4a68-a010-c50a01287fa3\") " pod="kuadrant-system/authorino-58cd955675-mbrt5"
Jul 25 01:21:23.294207 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.293953    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mbrt5"
Jul 25 01:21:23.329486 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.329111    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrmrf\" (UniqueName: \"kubernetes.io/projected/a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952-kube-api-access-zrmrf\") pod \"authorino-75d556bd77-n7s2d\" (UID: \"a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952\") " pod="kuadrant-system/authorino-75d556bd77-n7s2d"
Jul 25 01:21:23.337843 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.337789    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrmrf\" (UniqueName: \"kubernetes.io/projected/a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952-kube-api-access-zrmrf\") pod \"authorino-75d556bd77-n7s2d\" (UID: \"a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952\") " pod="kuadrant-system/authorino-75d556bd77-n7s2d"
Jul 25 01:21:23.473765 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.473519    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-n7s2d"
Jul 25 01:21:23.486053 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.484682    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-mbrt5"]
Jul 25 01:21:23.490208 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:21:23.490178    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d31e1bf_8738_4a68_a010_c50a01287fa3.slice/crio-02d3731d7bf6f5ab623afbffb8ae3df03a96fa4e95b574a115701bf5f7a99c52 WatchSource:0}: Error finding container 02d3731d7bf6f5ab623afbffb8ae3df03a96fa4e95b574a115701bf5f7a99c52: Status 404 returned error can't find the container with id 02d3731d7bf6f5ab623afbffb8ae3df03a96fa4e95b574a115701bf5f7a99c52
Jul 25 01:21:23.631241 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.631210    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-n7s2d"]
Jul 25 01:21:23.633313 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:21:23.633249    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda84d0e5a_7d8f_4f2f_b4a4_7e02776a8952.slice/crio-630bcc110ff0e37c70408f652a6103594fe3d6dd79b1d67abd6f3a8d39e2aa86 WatchSource:0}: Error finding container 630bcc110ff0e37c70408f652a6103594fe3d6dd79b1d67abd6f3a8d39e2aa86: Status 404 returned error can't find the container with id 630bcc110ff0e37c70408f652a6103594fe3d6dd79b1d67abd6f3a8d39e2aa86
Jul 25 01:21:23.780733 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.780692    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-n7s2d" event={"ID":"a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952","Type":"ContainerStarted","Data":"630bcc110ff0e37c70408f652a6103594fe3d6dd79b1d67abd6f3a8d39e2aa86"}
Jul 25 01:21:23.781971 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:23.781946    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mbrt5" event={"ID":"9d31e1bf-8738-4a68-a010-c50a01287fa3","Type":"ContainerStarted","Data":"02d3731d7bf6f5ab623afbffb8ae3df03a96fa4e95b574a115701bf5f7a99c52"}
Jul 25 01:21:26.804103 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:26.804064    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md" event={"ID":"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2","Type":"ContainerStarted","Data":"cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50"}
Jul 25 01:21:26.804520 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:26.804211    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:26.820109 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:26.820045    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md" podStartSLOduration=1.5963385460000001 podStartE2EDuration="4.820030636s" totalImagesPullingTime="3.22369209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:21:26.819401272 +0000 UTC m=+570.929210557" watchObservedRunningTime="2026-07-25 01:21:26.820030636 +0000 UTC m=+570.929839920"
Jul 25 01:21:27.812882 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:27.812852    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-n7s2d" event={"ID":"a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952","Type":"ContainerStarted","Data":"3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb"}
Jul 25 01:21:27.827148 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:27.827098    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-n7s2d" podStartSLOduration=0.726015361 podStartE2EDuration="4.827082942s" totalImagesPullingTime="4.101067581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:21:27.825695865 +0000 UTC m=+571.935505150" watchObservedRunningTime="2026-07-25 01:21:27.827082942 +0000 UTC m=+571.936892204"
Jul 25 01:21:27.838425 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:27.838375    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-mbrt5" podStartSLOduration=1.583254535 podStartE2EDuration="5.838357173s" totalImagesPullingTime="4.255102638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:21:27.837826212 +0000 UTC m=+571.947635587" watchObservedRunningTime="2026-07-25 01:21:27.838357173 +0000 UTC m=+571.948166458"
Jul 25 01:21:27.872590 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:27.872548    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-mbrt5"]
Jul 25 01:21:28.820117 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:28.820073    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mbrt5" event={"ID":"9d31e1bf-8738-4a68-a010-c50a01287fa3","Type":"ContainerStarted","Data":"1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6"}
Jul 25 01:21:28.820631 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:28.820404    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-mbrt5" podUID="9d31e1bf-8738-4a68-a010-c50a01287fa3" containerName="authorino" containerID="cri-o://1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6" gracePeriod=30
Jul 25 01:21:29.057092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.057065    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mbrt5"
Jul 25 01:21:29.182434 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.182407    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmwm8\" (UniqueName: \"kubernetes.io/projected/9d31e1bf-8738-4a68-a010-c50a01287fa3-kube-api-access-cmwm8\") pod \"9d31e1bf-8738-4a68-a010-c50a01287fa3\" (UID: \"9d31e1bf-8738-4a68-a010-c50a01287fa3\") "
Jul 25 01:21:29.184436 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.184405    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d31e1bf-8738-4a68-a010-c50a01287fa3-kube-api-access-cmwm8" (OuterVolumeSpecName: "kube-api-access-cmwm8") pod "9d31e1bf-8738-4a68-a010-c50a01287fa3" (UID: "9d31e1bf-8738-4a68-a010-c50a01287fa3"). InnerVolumeSpecName "kube-api-access-cmwm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:21:29.283214 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.283183    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmwm8\" (UniqueName: \"kubernetes.io/projected/9d31e1bf-8738-4a68-a010-c50a01287fa3-kube-api-access-cmwm8\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:21:29.825712 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.825675    2572 generic.go:356] "Generic (PLEG): container finished" podID="9d31e1bf-8738-4a68-a010-c50a01287fa3" containerID="1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6" exitCode=0
Jul 25 01:21:29.826132 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.825720    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-mbrt5"
Jul 25 01:21:29.826132 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.825763    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mbrt5" event={"ID":"9d31e1bf-8738-4a68-a010-c50a01287fa3","Type":"ContainerDied","Data":"1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6"}
Jul 25 01:21:29.826132 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.825809    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-mbrt5" event={"ID":"9d31e1bf-8738-4a68-a010-c50a01287fa3","Type":"ContainerDied","Data":"02d3731d7bf6f5ab623afbffb8ae3df03a96fa4e95b574a115701bf5f7a99c52"}
Jul 25 01:21:29.826132 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.825830    2572 scope.go:117] "RemoveContainer" containerID="1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6"
Jul 25 01:21:29.835021 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.835001    2572 scope.go:117] "RemoveContainer" containerID="1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6"
Jul 25 01:21:29.835271 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:21:29.835252    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6\": container with ID starting with 1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6 not found: ID does not exist" containerID="1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6"
Jul 25 01:21:29.835319 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.835279    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6"} err="failed to get container status \"1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6\": rpc error: code = NotFound desc = could not find container \"1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6\": container with ID starting with 1269bf379a440d36267d76fb8f7d81bb5aaf8496051d742b73e3a76fd532e6a6 not found: ID does not exist"
Jul 25 01:21:29.846495 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.846469    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-mbrt5"]
Jul 25 01:21:29.850187 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:29.850161    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-mbrt5"]
Jul 25 01:21:30.353351 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:30.353290    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d31e1bf-8738-4a68-a010-c50a01287fa3" path="/var/lib/kubelet/pods/9d31e1bf-8738-4a68-a010-c50a01287fa3/volumes"
Jul 25 01:21:36.669152 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:36.669115    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k45md"]
Jul 25 01:21:36.669628 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:36.669308    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md" podUID="6a4cbca7-1a77-45bd-a493-fb8364b1d1f2" containerName="limitador" containerID="cri-o://cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50" gracePeriod=30
Jul 25 01:21:36.670026 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:36.670008    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:37.000050 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:21:37.000026    2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4cbca7_1a77_45bd_a493_fb8364b1d1f2.slice/crio-cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50.scope\": RecentStats: unable to find data in memory cache]"
Jul 25 01:21:37.612210 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.612185    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:37.750816 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.750734    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86scv\" (UniqueName: \"kubernetes.io/projected/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-kube-api-access-86scv\") pod \"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2\" (UID: \"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2\") "
Jul 25 01:21:37.750816 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.750782    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-config-file\") pod \"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2\" (UID: \"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2\") "
Jul 25 01:21:37.751210 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.751119    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-config-file" (OuterVolumeSpecName: "config-file") pod "6a4cbca7-1a77-45bd-a493-fb8364b1d1f2" (UID: "6a4cbca7-1a77-45bd-a493-fb8364b1d1f2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:21:37.752819 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.752790    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-kube-api-access-86scv" (OuterVolumeSpecName: "kube-api-access-86scv") pod "6a4cbca7-1a77-45bd-a493-fb8364b1d1f2" (UID: "6a4cbca7-1a77-45bd-a493-fb8364b1d1f2"). InnerVolumeSpecName "kube-api-access-86scv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:21:37.852255 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.852212    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86scv\" (UniqueName: \"kubernetes.io/projected/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-kube-api-access-86scv\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:21:37.852255 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.852250    2572 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2-config-file\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:21:37.864424 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.864387    2572 generic.go:356] "Generic (PLEG): container finished" podID="6a4cbca7-1a77-45bd-a493-fb8364b1d1f2" containerID="cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50" exitCode=0
Jul 25 01:21:37.864555 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.864437    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md"
Jul 25 01:21:37.864555 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.864474    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md" event={"ID":"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2","Type":"ContainerDied","Data":"cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50"}
Jul 25 01:21:37.864555 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.864525    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-k45md" event={"ID":"6a4cbca7-1a77-45bd-a493-fb8364b1d1f2","Type":"ContainerDied","Data":"b2d504ddb0f751fa0d79b1ec30151341289ddb5cca70a608628e0a4053abcf83"}
Jul 25 01:21:37.864555 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.864548    2572 scope.go:117] "RemoveContainer" containerID="cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50"
Jul 25 01:21:37.873497 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.873480    2572 scope.go:117] "RemoveContainer" containerID="cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50"
Jul 25 01:21:37.873758 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:21:37.873736    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50\": container with ID starting with cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50 not found: ID does not exist" containerID="cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50"
Jul 25 01:21:37.873842 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.873762    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50"} err="failed to get container status \"cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50\": rpc error: code = NotFound desc = could not find container \"cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50\": container with ID starting with cc78b1298c0b2a0a684f2cd705dffd46f31c638898280de66434f0a7d64e4a50 not found: ID does not exist"
Jul 25 01:21:37.885275 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.885254    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k45md"]
Jul 25 01:21:37.889347 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:37.889312    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-k45md"]
Jul 25 01:21:38.357880 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:38.357847    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a4cbca7-1a77-45bd-a493-fb8364b1d1f2" path="/var/lib/kubelet/pods/6a4cbca7-1a77-45bd-a493-fb8364b1d1f2/volumes"
Jul 25 01:21:42.365513 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.365477    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"]
Jul 25 01:21:42.365970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.365782    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a4cbca7-1a77-45bd-a493-fb8364b1d1f2" containerName="limitador"
Jul 25 01:21:42.365970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.365793    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a4cbca7-1a77-45bd-a493-fb8364b1d1f2" containerName="limitador"
Jul 25 01:21:42.365970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.365814    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d31e1bf-8738-4a68-a010-c50a01287fa3" containerName="authorino"
Jul 25 01:21:42.365970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.365820    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d31e1bf-8738-4a68-a010-c50a01287fa3" containerName="authorino"
Jul 25 01:21:42.365970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.365866    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d31e1bf-8738-4a68-a010-c50a01287fa3" containerName="authorino"
Jul 25 01:21:42.365970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.365874    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a4cbca7-1a77-45bd-a493-fb8364b1d1f2" containerName="limitador"
Jul 25 01:21:42.370138 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.370117    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"
Jul 25 01:21:42.372838 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.372818    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 25 01:21:42.372941 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.372884    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 25 01:21:42.372941 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.372893    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 25 01:21:42.373683 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.373668    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9gdmk\""
Jul 25 01:21:42.377279 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.377044    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"]
Jul 25 01:21:42.490400 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.490362    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a300f37a-5848-4633-bb57-36bd7c2c5b3b-data\") pod \"postgres-69fb4594cf-lshxp\" (UID: \"a300f37a-5848-4633-bb57-36bd7c2c5b3b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"
Jul 25 01:21:42.490568 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.490427    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsvjj\" (UniqueName: \"kubernetes.io/projected/a300f37a-5848-4633-bb57-36bd7c2c5b3b-kube-api-access-rsvjj\") pod \"postgres-69fb4594cf-lshxp\" (UID: \"a300f37a-5848-4633-bb57-36bd7c2c5b3b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"
Jul 25 01:21:42.591798 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.591768    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a300f37a-5848-4633-bb57-36bd7c2c5b3b-data\") pod \"postgres-69fb4594cf-lshxp\" (UID: \"a300f37a-5848-4633-bb57-36bd7c2c5b3b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"
Jul 25 01:21:42.591968 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.591821    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsvjj\" (UniqueName: \"kubernetes.io/projected/a300f37a-5848-4633-bb57-36bd7c2c5b3b-kube-api-access-rsvjj\") pod \"postgres-69fb4594cf-lshxp\" (UID: \"a300f37a-5848-4633-bb57-36bd7c2c5b3b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"
Jul 25 01:21:42.592199 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.592172    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a300f37a-5848-4633-bb57-36bd7c2c5b3b-data\") pod \"postgres-69fb4594cf-lshxp\" (UID: \"a300f37a-5848-4633-bb57-36bd7c2c5b3b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"
Jul 25 01:21:42.600047 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.600022    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsvjj\" (UniqueName: \"kubernetes.io/projected/a300f37a-5848-4633-bb57-36bd7c2c5b3b-kube-api-access-rsvjj\") pod \"postgres-69fb4594cf-lshxp\" (UID: \"a300f37a-5848-4633-bb57-36bd7c2c5b3b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"
Jul 25 01:21:42.681789 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.681759    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"
Jul 25 01:21:42.812036 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.812010    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"]
Jul 25 01:21:42.814289 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:21:42.814262    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda300f37a_5848_4633_bb57_36bd7c2c5b3b.slice/crio-6953385e223b1998f8143676b5a51ffd883f1813f39149ebbbb478eebcafeabc WatchSource:0}: Error finding container 6953385e223b1998f8143676b5a51ffd883f1813f39149ebbbb478eebcafeabc: Status 404 returned error can't find the container with id 6953385e223b1998f8143676b5a51ffd883f1813f39149ebbbb478eebcafeabc
Jul 25 01:21:42.889410 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:42.889377    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp" event={"ID":"a300f37a-5848-4633-bb57-36bd7c2c5b3b","Type":"ContainerStarted","Data":"6953385e223b1998f8143676b5a51ffd883f1813f39149ebbbb478eebcafeabc"}
Jul 25 01:21:48.921945 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:48.921916    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp" event={"ID":"a300f37a-5848-4633-bb57-36bd7c2c5b3b","Type":"ContainerStarted","Data":"f34b85e3fe6315de0dd7bd6ecb3e897766906e3a7239394372685882eefe652e"}
Jul 25 01:21:48.922258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:48.922045    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"
Jul 25 01:21:48.937384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:48.937334    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp" podStartSLOduration=0.905783251 podStartE2EDuration="6.937307559s" totalImagesPullingTime="6.031524308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:21:48.937187007 +0000 UTC m=+593.046996310" watchObservedRunningTime="2026-07-25 01:21:48.937307559 +0000 UTC m=+593.047116843"
Jul 25 01:21:54.954990 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:54.954959    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lshxp"
Jul 25 01:21:55.493264 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.493228    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-f8nsg"]
Jul 25 01:21:55.501978 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.501952    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-f8nsg"
Jul 25 01:21:55.503131 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.503108    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-f8nsg"]
Jul 25 01:21:55.603048 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.603014    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77kzg\" (UniqueName: \"kubernetes.io/projected/7dbea0d1-b963-461f-be1b-2f395d78f13c-kube-api-access-77kzg\") pod \"authorino-5d4b67c446-f8nsg\" (UID: \"7dbea0d1-b963-461f-be1b-2f395d78f13c\") " pod="kuadrant-system/authorino-5d4b67c446-f8nsg"
Jul 25 01:21:55.674314 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.674279    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-f8nsg"]
Jul 25 01:21:55.674542 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:21:55.674521    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-77kzg], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-f8nsg" podUID="7dbea0d1-b963-461f-be1b-2f395d78f13c"
Jul 25 01:21:55.699931 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.699907    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-qwgnb"]
Jul 25 01:21:55.703818 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.703797    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qwgnb"
Jul 25 01:21:55.704003 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.703979    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77kzg\" (UniqueName: \"kubernetes.io/projected/7dbea0d1-b963-461f-be1b-2f395d78f13c-kube-api-access-77kzg\") pod \"authorino-5d4b67c446-f8nsg\" (UID: \"7dbea0d1-b963-461f-be1b-2f395d78f13c\") " pod="kuadrant-system/authorino-5d4b67c446-f8nsg"
Jul 25 01:21:55.706045 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.706026    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 25 01:21:55.711957 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.711938    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-qwgnb"]
Jul 25 01:21:55.719117 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.719097    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77kzg\" (UniqueName: \"kubernetes.io/projected/7dbea0d1-b963-461f-be1b-2f395d78f13c-kube-api-access-77kzg\") pod \"authorino-5d4b67c446-f8nsg\" (UID: \"7dbea0d1-b963-461f-be1b-2f395d78f13c\") " pod="kuadrant-system/authorino-5d4b67c446-f8nsg"
Jul 25 01:21:55.749565 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.749514    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-qwgnb"]
Jul 25 01:21:55.749739 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:21:55.749721    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-stk85 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-stk85 tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-qwgnb" podUID="6f23ee04-9c02-422e-af24-fe5d59465608"
Jul 25 01:21:55.783812 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.783781    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85c4b6677c-pbkt9"]
Jul 25 01:21:55.787466 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.787449    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85c4b6677c-pbkt9"
Jul 25 01:21:55.793877 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.793853    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85c4b6677c-pbkt9"]
Jul 25 01:21:55.805073 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.805048    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stk85\" (UniqueName: \"kubernetes.io/projected/6f23ee04-9c02-422e-af24-fe5d59465608-kube-api-access-stk85\") pod \"authorino-76877994-qwgnb\" (UID: \"6f23ee04-9c02-422e-af24-fe5d59465608\") " pod="kuadrant-system/authorino-76877994-qwgnb"
Jul 25 01:21:55.805168 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.805098    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f23ee04-9c02-422e-af24-fe5d59465608-tls-cert\") pod \"authorino-76877994-qwgnb\" (UID: \"6f23ee04-9c02-422e-af24-fe5d59465608\") " pod="kuadrant-system/authorino-76877994-qwgnb"
Jul 25 01:21:55.805168 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.805129    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvnd8\" (UniqueName: \"kubernetes.io/projected/3579d459-80a1-4974-b2e7-a3231897b4cf-kube-api-access-kvnd8\") pod \"authorino-85c4b6677c-pbkt9\" (UID: \"3579d459-80a1-4974-b2e7-a3231897b4cf\") " pod="kuadrant-system/authorino-85c4b6677c-pbkt9"
Jul 25 01:21:55.805168 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.805152    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3579d459-80a1-4974-b2e7-a3231897b4cf-tls-cert\") pod \"authorino-85c4b6677c-pbkt9\" (UID: \"3579d459-80a1-4974-b2e7-a3231897b4cf\") " pod="kuadrant-system/authorino-85c4b6677c-pbkt9"
Jul 25 01:21:55.906072 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.906040    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f23ee04-9c02-422e-af24-fe5d59465608-tls-cert\") pod \"authorino-76877994-qwgnb\" (UID: \"6f23ee04-9c02-422e-af24-fe5d59465608\") " pod="kuadrant-system/authorino-76877994-qwgnb"
Jul 25 01:21:55.906226 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.906088    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvnd8\" (UniqueName: \"kubernetes.io/projected/3579d459-80a1-4974-b2e7-a3231897b4cf-kube-api-access-kvnd8\") pod \"authorino-85c4b6677c-pbkt9\" (UID: \"3579d459-80a1-4974-b2e7-a3231897b4cf\") " pod="kuadrant-system/authorino-85c4b6677c-pbkt9"
Jul 25 01:21:55.906226 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.906115    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3579d459-80a1-4974-b2e7-a3231897b4cf-tls-cert\") pod \"authorino-85c4b6677c-pbkt9\" (UID: \"3579d459-80a1-4974-b2e7-a3231897b4cf\") " pod="kuadrant-system/authorino-85c4b6677c-pbkt9"
Jul 25 01:21:55.906226 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.906140    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stk85\" (UniqueName: \"kubernetes.io/projected/6f23ee04-9c02-422e-af24-fe5d59465608-kube-api-access-stk85\") pod \"authorino-76877994-qwgnb\" (UID: \"6f23ee04-9c02-422e-af24-fe5d59465608\") " pod="kuadrant-system/authorino-76877994-qwgnb"
Jul 25 01:21:55.908718 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.908692    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3579d459-80a1-4974-b2e7-a3231897b4cf-tls-cert\") pod \"authorino-85c4b6677c-pbkt9\" (UID: \"3579d459-80a1-4974-b2e7-a3231897b4cf\") " pod="kuadrant-system/authorino-85c4b6677c-pbkt9"
Jul 25 01:21:55.908822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.908692    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f23ee04-9c02-422e-af24-fe5d59465608-tls-cert\") pod \"authorino-76877994-qwgnb\" (UID: \"6f23ee04-9c02-422e-af24-fe5d59465608\") " pod="kuadrant-system/authorino-76877994-qwgnb"
Jul 25 01:21:55.913349 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.913307    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvnd8\" (UniqueName: \"kubernetes.io/projected/3579d459-80a1-4974-b2e7-a3231897b4cf-kube-api-access-kvnd8\") pod \"authorino-85c4b6677c-pbkt9\" (UID: \"3579d459-80a1-4974-b2e7-a3231897b4cf\") " pod="kuadrant-system/authorino-85c4b6677c-pbkt9"
Jul 25 01:21:55.913608 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.913586    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stk85\" (UniqueName: \"kubernetes.io/projected/6f23ee04-9c02-422e-af24-fe5d59465608-kube-api-access-stk85\") pod \"authorino-76877994-qwgnb\" (UID: \"6f23ee04-9c02-422e-af24-fe5d59465608\") " pod="kuadrant-system/authorino-76877994-qwgnb"
Jul 25 01:21:55.953916 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.953883    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-f8nsg"
Jul 25 01:21:55.954071 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.953883    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qwgnb"
Jul 25 01:21:55.961535 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.961514    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-f8nsg"
Jul 25 01:21:55.964145 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:55.964124    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qwgnb"
Jul 25 01:21:56.006459 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.006398    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f23ee04-9c02-422e-af24-fe5d59465608-tls-cert\") pod \"6f23ee04-9c02-422e-af24-fe5d59465608\" (UID: \"6f23ee04-9c02-422e-af24-fe5d59465608\") "
Jul 25 01:21:56.006459 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.006449    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stk85\" (UniqueName: \"kubernetes.io/projected/6f23ee04-9c02-422e-af24-fe5d59465608-kube-api-access-stk85\") pod \"6f23ee04-9c02-422e-af24-fe5d59465608\" (UID: \"6f23ee04-9c02-422e-af24-fe5d59465608\") "
Jul 25 01:21:56.006599 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.006478    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77kzg\" (UniqueName: \"kubernetes.io/projected/7dbea0d1-b963-461f-be1b-2f395d78f13c-kube-api-access-77kzg\") pod \"7dbea0d1-b963-461f-be1b-2f395d78f13c\" (UID: \"7dbea0d1-b963-461f-be1b-2f395d78f13c\") "
Jul 25 01:21:56.008457 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.008425    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f23ee04-9c02-422e-af24-fe5d59465608-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6f23ee04-9c02-422e-af24-fe5d59465608" (UID: "6f23ee04-9c02-422e-af24-fe5d59465608"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:21:56.008552 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.008501    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbea0d1-b963-461f-be1b-2f395d78f13c-kube-api-access-77kzg" (OuterVolumeSpecName: "kube-api-access-77kzg") pod "7dbea0d1-b963-461f-be1b-2f395d78f13c" (UID: "7dbea0d1-b963-461f-be1b-2f395d78f13c"). InnerVolumeSpecName "kube-api-access-77kzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:21:56.008596 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.008545    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f23ee04-9c02-422e-af24-fe5d59465608-kube-api-access-stk85" (OuterVolumeSpecName: "kube-api-access-stk85") pod "6f23ee04-9c02-422e-af24-fe5d59465608" (UID: "6f23ee04-9c02-422e-af24-fe5d59465608"). InnerVolumeSpecName "kube-api-access-stk85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:21:56.097487 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.097452    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85c4b6677c-pbkt9"
Jul 25 01:21:56.107164 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.107139    2572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f23ee04-9c02-422e-af24-fe5d59465608-tls-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:21:56.107164 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.107162    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stk85\" (UniqueName: \"kubernetes.io/projected/6f23ee04-9c02-422e-af24-fe5d59465608-kube-api-access-stk85\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:21:56.107333 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.107173    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77kzg\" (UniqueName: \"kubernetes.io/projected/7dbea0d1-b963-461f-be1b-2f395d78f13c-kube-api-access-77kzg\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:21:56.223306 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.223275    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85c4b6677c-pbkt9"]
Jul 25 01:21:56.226536 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:21:56.226508    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3579d459_80a1_4974_b2e7_a3231897b4cf.slice/crio-2555e9c77cfb598b2ac9eb4ae01d1d5e116664d6119acd4a120f4ad993a0fa55 WatchSource:0}: Error finding container 2555e9c77cfb598b2ac9eb4ae01d1d5e116664d6119acd4a120f4ad993a0fa55: Status 404 returned error can't find the container with id 2555e9c77cfb598b2ac9eb4ae01d1d5e116664d6119acd4a120f4ad993a0fa55
Jul 25 01:21:56.959661 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.959623    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85c4b6677c-pbkt9" event={"ID":"3579d459-80a1-4974-b2e7-a3231897b4cf","Type":"ContainerStarted","Data":"49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460"}
Jul 25 01:21:56.959836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.959672    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85c4b6677c-pbkt9" event={"ID":"3579d459-80a1-4974-b2e7-a3231897b4cf","Type":"ContainerStarted","Data":"2555e9c77cfb598b2ac9eb4ae01d1d5e116664d6119acd4a120f4ad993a0fa55"}
Jul 25 01:21:56.959836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.959681    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qwgnb"
Jul 25 01:21:56.959934 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.959919    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-f8nsg"
Jul 25 01:21:56.976779 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.976726    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85c4b6677c-pbkt9" podStartSLOduration=1.691720578 podStartE2EDuration="1.976713399s" totalImagesPullingTime="284.992821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:21:56.974738667 +0000 UTC m=+601.084547952" watchObservedRunningTime="2026-07-25 01:21:56.976713399 +0000 UTC m=+601.086522682"
Jul 25 01:21:56.997518 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:56.997486    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-f8nsg"]
Jul 25 01:21:57.006058 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.005956    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-n7s2d"]
Jul 25 01:21:57.006209 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.006131    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-n7s2d" podUID="a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952" containerName="authorino" containerID="cri-o://3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb" gracePeriod=30
Jul 25 01:21:57.007804 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.007776    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-f8nsg"]
Jul 25 01:21:57.047704 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.047672    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-qwgnb"]
Jul 25 01:21:57.061369 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.061344    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-qwgnb"]
Jul 25 01:21:57.248142 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.248109    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-n7s2d"
Jul 25 01:21:57.318217 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.318184    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrmrf\" (UniqueName: \"kubernetes.io/projected/a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952-kube-api-access-zrmrf\") pod \"a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952\" (UID: \"a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952\") "
Jul 25 01:21:57.320259 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.320234    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952-kube-api-access-zrmrf" (OuterVolumeSpecName: "kube-api-access-zrmrf") pod "a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952" (UID: "a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952"). InnerVolumeSpecName "kube-api-access-zrmrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:21:57.419399 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.419365    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrmrf\" (UniqueName: \"kubernetes.io/projected/a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952-kube-api-access-zrmrf\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:21:57.965554 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.965512    2572 generic.go:356] "Generic (PLEG): container finished" podID="a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952" containerID="3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb" exitCode=0
Jul 25 01:21:57.965730 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.965570    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-n7s2d"
Jul 25 01:21:57.965730 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.965590    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-n7s2d" event={"ID":"a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952","Type":"ContainerDied","Data":"3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb"}
Jul 25 01:21:57.965730 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.965635    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-n7s2d" event={"ID":"a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952","Type":"ContainerDied","Data":"630bcc110ff0e37c70408f652a6103594fe3d6dd79b1d67abd6f3a8d39e2aa86"}
Jul 25 01:21:57.965730 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.965654    2572 scope.go:117] "RemoveContainer" containerID="3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb"
Jul 25 01:21:57.974730 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.974714    2572 scope.go:117] "RemoveContainer" containerID="3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb"
Jul 25 01:21:57.974981 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:21:57.974964    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb\": container with ID starting with 3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb not found: ID does not exist" containerID="3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb"
Jul 25 01:21:57.975017 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.974988    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb"} err="failed to get container status \"3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb\": rpc error: code = NotFound desc = could not find container \"3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb\": container with ID starting with 3984ad4f4254039e176b053e2065d3bc9cc44cec342e694162c316d8604c55bb not found: ID does not exist"
Jul 25 01:21:57.986286 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.986261    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-n7s2d"]
Jul 25 01:21:57.988005 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:57.987987    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-n7s2d"]
Jul 25 01:21:58.353067 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:58.352975    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f23ee04-9c02-422e-af24-fe5d59465608" path="/var/lib/kubelet/pods/6f23ee04-9c02-422e-af24-fe5d59465608/volumes"
Jul 25 01:21:58.353289 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:58.353271    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dbea0d1-b963-461f-be1b-2f395d78f13c" path="/var/lib/kubelet/pods/7dbea0d1-b963-461f-be1b-2f395d78f13c/volumes"
Jul 25 01:21:58.353549 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:21:58.353532    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952" path="/var/lib/kubelet/pods/a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952/volumes"
Jul 25 01:22:38.304216 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.304148    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"]
Jul 25 01:22:38.304799 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.304691    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952" containerName="authorino"
Jul 25 01:22:38.304799 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.304710    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952" containerName="authorino"
Jul 25 01:22:38.304799 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.304767    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a84d0e5a-7d8f-4f2f-b4a4-7e02776a8952" containerName="authorino"
Jul 25 01:22:38.307551 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.307535    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:38.310109 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.310089    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 25 01:22:38.318103 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.318081    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"]
Jul 25 01:22:38.454247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.454211    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvjf\" (UniqueName: \"kubernetes.io/projected/774f4ffa-4f01-49f0-ad82-49289fc182d8-kube-api-access-mmvjf\") pod \"maas-api-6d6c5bd49-js487\" (UID: \"774f4ffa-4f01-49f0-ad82-49289fc182d8\") " pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:38.454464 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.454263    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774f4ffa-4f01-49f0-ad82-49289fc182d8-openshift-service-ca\") pod \"maas-api-6d6c5bd49-js487\" (UID: \"774f4ffa-4f01-49f0-ad82-49289fc182d8\") " pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:38.454464 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.454377    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774f4ffa-4f01-49f0-ad82-49289fc182d8-maas-api-tls\") pod \"maas-api-6d6c5bd49-js487\" (UID: \"774f4ffa-4f01-49f0-ad82-49289fc182d8\") " pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:38.555562 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.555484    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvjf\" (UniqueName: \"kubernetes.io/projected/774f4ffa-4f01-49f0-ad82-49289fc182d8-kube-api-access-mmvjf\") pod \"maas-api-6d6c5bd49-js487\" (UID: \"774f4ffa-4f01-49f0-ad82-49289fc182d8\") " pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:38.555562 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.555518    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774f4ffa-4f01-49f0-ad82-49289fc182d8-openshift-service-ca\") pod \"maas-api-6d6c5bd49-js487\" (UID: \"774f4ffa-4f01-49f0-ad82-49289fc182d8\") " pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:38.555562 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.555555    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774f4ffa-4f01-49f0-ad82-49289fc182d8-maas-api-tls\") pod \"maas-api-6d6c5bd49-js487\" (UID: \"774f4ffa-4f01-49f0-ad82-49289fc182d8\") " pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:38.556225 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.556199    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/774f4ffa-4f01-49f0-ad82-49289fc182d8-openshift-service-ca\") pod \"maas-api-6d6c5bd49-js487\" (UID: \"774f4ffa-4f01-49f0-ad82-49289fc182d8\") " pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:38.557815 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.557789    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/774f4ffa-4f01-49f0-ad82-49289fc182d8-maas-api-tls\") pod \"maas-api-6d6c5bd49-js487\" (UID: \"774f4ffa-4f01-49f0-ad82-49289fc182d8\") " pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:38.563445 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.563418    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvjf\" (UniqueName: \"kubernetes.io/projected/774f4ffa-4f01-49f0-ad82-49289fc182d8-kube-api-access-mmvjf\") pod \"maas-api-6d6c5bd49-js487\" (UID: \"774f4ffa-4f01-49f0-ad82-49289fc182d8\") " pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:38.618208 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.618182    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:38.752463 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.752440    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"]
Jul 25 01:22:38.754094 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:22:38.754067    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774f4ffa_4f01_49f0_ad82_49289fc182d8.slice/crio-d51ce7b112264ce82453a8a0d7a9d2b6e1c9c20fe2955c0bcc4c18fa93399369 WatchSource:0}: Error finding container d51ce7b112264ce82453a8a0d7a9d2b6e1c9c20fe2955c0bcc4c18fa93399369: Status 404 returned error can't find the container with id d51ce7b112264ce82453a8a0d7a9d2b6e1c9c20fe2955c0bcc4c18fa93399369
Jul 25 01:22:38.755376 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:38.755355    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 25 01:22:39.158782 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:39.158742    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487" event={"ID":"774f4ffa-4f01-49f0-ad82-49289fc182d8","Type":"ContainerStarted","Data":"d51ce7b112264ce82453a8a0d7a9d2b6e1c9c20fe2955c0bcc4c18fa93399369"}
Jul 25 01:22:42.175862 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:42.175829    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487" event={"ID":"774f4ffa-4f01-49f0-ad82-49289fc182d8","Type":"ContainerStarted","Data":"f7f78f30f9420c8376930f4cd79d64a12371de5db58dcab4ce1e7083633ac266"}
Jul 25 01:22:42.176252 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:42.175888    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:22:42.192603 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:42.192555    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487" podStartSLOduration=1.7136914509999999 podStartE2EDuration="4.192538656s" totalImagesPullingTime="2.478847205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:22:42.190883878 +0000 UTC m=+646.300693156" watchObservedRunningTime="2026-07-25 01:22:42.192538656 +0000 UTC m=+646.302347939"
Jul 25 01:22:48.185428 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:22:48.185398    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d6c5bd49-js487"
Jul 25 01:23:06.297619 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.297583    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"]
Jul 25 01:23:06.304260 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.304243    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.306683 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.306661    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 25 01:23:06.306791 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.306704    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 25 01:23:06.307644 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.307622    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 25 01:23:06.307761 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.307687    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ls94p\""
Jul 25 01:23:06.310291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.310269    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"]
Jul 25 01:23:06.375580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.375552    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a58ad35-21f8-46d1-aee5-fc8267eba096-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.375723 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.375607    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.375723 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.375640    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nczh\" (UniqueName: \"kubernetes.io/projected/3a58ad35-21f8-46d1-aee5-fc8267eba096-kube-api-access-6nczh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.375723 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.375668    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.375723 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.375688    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.375857 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.375735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.478436 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.478391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.478436 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.478437    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nczh\" (UniqueName: \"kubernetes.io/projected/3a58ad35-21f8-46d1-aee5-fc8267eba096-kube-api-access-6nczh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.478667 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.478475    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.478667 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.478504    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.478667 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.478534    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.478667 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.478586    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a58ad35-21f8-46d1-aee5-fc8267eba096-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.478859 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.478807    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.479453 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.479431    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.479598 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.479509    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.481141 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.481117    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a58ad35-21f8-46d1-aee5-fc8267eba096-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.481376 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.481361    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a58ad35-21f8-46d1-aee5-fc8267eba096-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.485948 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.485922    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nczh\" (UniqueName: \"kubernetes.io/projected/3a58ad35-21f8-46d1-aee5-fc8267eba096-kube-api-access-6nczh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jl5fb\" (UID: \"3a58ad35-21f8-46d1-aee5-fc8267eba096\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.615104 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.615036    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:06.742930 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:06.742903    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"]
Jul 25 01:23:06.744625 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:23:06.744592    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a58ad35_21f8_46d1_aee5_fc8267eba096.slice/crio-073c2e3333e0d939498c3090f03c5f8afc118abcba0b031104b43dbc944f4657 WatchSource:0}: Error finding container 073c2e3333e0d939498c3090f03c5f8afc118abcba0b031104b43dbc944f4657: Status 404 returned error can't find the container with id 073c2e3333e0d939498c3090f03c5f8afc118abcba0b031104b43dbc944f4657
Jul 25 01:23:07.295759 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:07.295706    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb" event={"ID":"3a58ad35-21f8-46d1-aee5-fc8267eba096","Type":"ContainerStarted","Data":"073c2e3333e0d939498c3090f03c5f8afc118abcba0b031104b43dbc944f4657"}
Jul 25 01:23:08.301674 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:08.301635    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb" event={"ID":"3a58ad35-21f8-46d1-aee5-fc8267eba096","Type":"ContainerStarted","Data":"50a55dc5a64af32b677978f985016276934200402dd508244ead44d413a73741"}
Jul 25 01:23:08.317853 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:08.317806    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb" podStartSLOduration=1.1994439510000001 podStartE2EDuration="2.317788063s" totalImagesPullingTime="1.118344112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:23:08.317155622 +0000 UTC m=+672.426964905" watchObservedRunningTime="2026-07-25 01:23:08.317788063 +0000 UTC m=+672.427597398"
Jul 25 01:23:16.616050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:16.616016    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:16.616519 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:16.616075    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:16.628544 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:16.628523    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:23:17.364042 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:23:17.364007    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jl5fb"
Jul 25 01:24:55.979111 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:55.979076    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cf5d9ddd-x6svt"]
Jul 25 01:24:55.982293 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:55.982277    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf5d9ddd-x6svt"
Jul 25 01:24:55.988154 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:55.988126    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cf5d9ddd-x6svt"]
Jul 25 01:24:56.086171 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:56.086140    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh29r\" (UniqueName: \"kubernetes.io/projected/6054bfae-2e7a-4f2c-a37c-a35845d024d6-kube-api-access-gh29r\") pod \"authorino-6cf5d9ddd-x6svt\" (UID: \"6054bfae-2e7a-4f2c-a37c-a35845d024d6\") " pod="kuadrant-system/authorino-6cf5d9ddd-x6svt"
Jul 25 01:24:56.086317 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:56.086193    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6054bfae-2e7a-4f2c-a37c-a35845d024d6-tls-cert\") pod \"authorino-6cf5d9ddd-x6svt\" (UID: \"6054bfae-2e7a-4f2c-a37c-a35845d024d6\") " pod="kuadrant-system/authorino-6cf5d9ddd-x6svt"
Jul 25 01:24:56.186851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:56.186820    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh29r\" (UniqueName: \"kubernetes.io/projected/6054bfae-2e7a-4f2c-a37c-a35845d024d6-kube-api-access-gh29r\") pod \"authorino-6cf5d9ddd-x6svt\" (UID: \"6054bfae-2e7a-4f2c-a37c-a35845d024d6\") " pod="kuadrant-system/authorino-6cf5d9ddd-x6svt"
Jul 25 01:24:56.186984 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:56.186873    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6054bfae-2e7a-4f2c-a37c-a35845d024d6-tls-cert\") pod \"authorino-6cf5d9ddd-x6svt\" (UID: \"6054bfae-2e7a-4f2c-a37c-a35845d024d6\") " pod="kuadrant-system/authorino-6cf5d9ddd-x6svt"
Jul 25 01:24:56.189205 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:56.189183    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6054bfae-2e7a-4f2c-a37c-a35845d024d6-tls-cert\") pod \"authorino-6cf5d9ddd-x6svt\" (UID: \"6054bfae-2e7a-4f2c-a37c-a35845d024d6\") " pod="kuadrant-system/authorino-6cf5d9ddd-x6svt"
Jul 25 01:24:56.193675 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:56.193651    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh29r\" (UniqueName: \"kubernetes.io/projected/6054bfae-2e7a-4f2c-a37c-a35845d024d6-kube-api-access-gh29r\") pod \"authorino-6cf5d9ddd-x6svt\" (UID: \"6054bfae-2e7a-4f2c-a37c-a35845d024d6\") " pod="kuadrant-system/authorino-6cf5d9ddd-x6svt"
Jul 25 01:24:56.292763 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:56.292693    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf5d9ddd-x6svt"
Jul 25 01:24:56.425293 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:56.425268    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cf5d9ddd-x6svt"]
Jul 25 01:24:56.427063 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:24:56.427034    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6054bfae_2e7a_4f2c_a37c_a35845d024d6.slice/crio-da58d395dfb187d50e17330bb1f94c825fc29345d982921882357a2655ab1494 WatchSource:0}: Error finding container da58d395dfb187d50e17330bb1f94c825fc29345d982921882357a2655ab1494: Status 404 returned error can't find the container with id da58d395dfb187d50e17330bb1f94c825fc29345d982921882357a2655ab1494
Jul 25 01:24:56.826045 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:56.826014    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf5d9ddd-x6svt" event={"ID":"6054bfae-2e7a-4f2c-a37c-a35845d024d6","Type":"ContainerStarted","Data":"da58d395dfb187d50e17330bb1f94c825fc29345d982921882357a2655ab1494"}
Jul 25 01:24:57.832778 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:57.832746    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf5d9ddd-x6svt" event={"ID":"6054bfae-2e7a-4f2c-a37c-a35845d024d6","Type":"ContainerStarted","Data":"019f7dd87f82999485238edd1c2819d97224bc3459e540dd48e156d77e4e7767"}
Jul 25 01:24:57.850960 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:57.850892    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cf5d9ddd-x6svt" podStartSLOduration=2.487955853 podStartE2EDuration="2.850871521s" totalImagesPullingTime="362.915668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:24:57.847734782 +0000 UTC m=+781.957544077" watchObservedRunningTime="2026-07-25 01:24:57.850871521 +0000 UTC m=+781.960680806"
Jul 25 01:24:57.883004 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:57.882965    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85c4b6677c-pbkt9"]
Jul 25 01:24:57.883183 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:57.883155    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-85c4b6677c-pbkt9" podUID="3579d459-80a1-4974-b2e7-a3231897b4cf" containerName="authorino" containerID="cri-o://49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460" gracePeriod=30
Jul 25 01:24:58.151886 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.151859    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85c4b6677c-pbkt9"
Jul 25 01:24:58.305566 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.305531    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3579d459-80a1-4974-b2e7-a3231897b4cf-tls-cert\") pod \"3579d459-80a1-4974-b2e7-a3231897b4cf\" (UID: \"3579d459-80a1-4974-b2e7-a3231897b4cf\") "
Jul 25 01:24:58.305787 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.305586    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvnd8\" (UniqueName: \"kubernetes.io/projected/3579d459-80a1-4974-b2e7-a3231897b4cf-kube-api-access-kvnd8\") pod \"3579d459-80a1-4974-b2e7-a3231897b4cf\" (UID: \"3579d459-80a1-4974-b2e7-a3231897b4cf\") "
Jul 25 01:24:58.307645 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.307607    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3579d459-80a1-4974-b2e7-a3231897b4cf-kube-api-access-kvnd8" (OuterVolumeSpecName: "kube-api-access-kvnd8") pod "3579d459-80a1-4974-b2e7-a3231897b4cf" (UID: "3579d459-80a1-4974-b2e7-a3231897b4cf"). InnerVolumeSpecName "kube-api-access-kvnd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:24:58.316597 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.316562    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3579d459-80a1-4974-b2e7-a3231897b4cf-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3579d459-80a1-4974-b2e7-a3231897b4cf" (UID: "3579d459-80a1-4974-b2e7-a3231897b4cf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:24:58.406282 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.406247    2572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3579d459-80a1-4974-b2e7-a3231897b4cf-tls-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:24:58.406282 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.406283    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvnd8\" (UniqueName: \"kubernetes.io/projected/3579d459-80a1-4974-b2e7-a3231897b4cf-kube-api-access-kvnd8\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:24:58.838886 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.838809    2572 generic.go:356] "Generic (PLEG): container finished" podID="3579d459-80a1-4974-b2e7-a3231897b4cf" containerID="49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460" exitCode=0
Jul 25 01:24:58.838886 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.838852    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85c4b6677c-pbkt9"
Jul 25 01:24:58.839311 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.838891    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85c4b6677c-pbkt9" event={"ID":"3579d459-80a1-4974-b2e7-a3231897b4cf","Type":"ContainerDied","Data":"49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460"}
Jul 25 01:24:58.839311 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.838929    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85c4b6677c-pbkt9" event={"ID":"3579d459-80a1-4974-b2e7-a3231897b4cf","Type":"ContainerDied","Data":"2555e9c77cfb598b2ac9eb4ae01d1d5e116664d6119acd4a120f4ad993a0fa55"}
Jul 25 01:24:58.839311 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.838945    2572 scope.go:117] "RemoveContainer" containerID="49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460"
Jul 25 01:24:58.847366 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.847342    2572 scope.go:117] "RemoveContainer" containerID="49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460"
Jul 25 01:24:58.847643 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:24:58.847625    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460\": container with ID starting with 49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460 not found: ID does not exist" containerID="49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460"
Jul 25 01:24:58.847709 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.847656    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460"} err="failed to get container status \"49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460\": rpc error: code = NotFound desc = could not find container \"49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460\": container with ID starting with 49f2697120c008192b6c2363e12260e7f847800df63ab8547cd4cf240796d460 not found: ID does not exist"
Jul 25 01:24:58.859846 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.859823    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-85c4b6677c-pbkt9"]
Jul 25 01:24:58.862963 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:24:58.862941    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-85c4b6677c-pbkt9"]
Jul 25 01:25:00.352844 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:25:00.352807    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3579d459-80a1-4974-b2e7-a3231897b4cf" path="/var/lib/kubelet/pods/3579d459-80a1-4974-b2e7-a3231897b4cf/volumes"
Jul 25 01:30:00.158841 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:00.158801    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29749050-pqr5s"]
Jul 25 01:30:00.161206 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:00.159133    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3579d459-80a1-4974-b2e7-a3231897b4cf" containerName="authorino"
Jul 25 01:30:00.161206 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:00.159145    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3579d459-80a1-4974-b2e7-a3231897b4cf" containerName="authorino"
Jul 25 01:30:00.161206 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:00.159211    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3579d459-80a1-4974-b2e7-a3231897b4cf" containerName="authorino"
Jul 25 01:30:00.162055 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:00.162040    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749050-pqr5s"
Jul 25 01:30:00.164830 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:00.164809    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 01:30:00.171002 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:00.170985    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749050-pqr5s"
Jul 25 01:30:00.177609 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:00.177579    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29749050-pqr5s"]
Jul 25 01:30:00.304947 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:00.304923    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29749050-pqr5s"]
Jul 25 01:30:00.306944 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:30:00.306914    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0cd0ad_e1ed_4788_b244_16dface55598.slice/crio-a3be40652bd724f348718337c599e3c5c5d400adabe0582742bbbec2c098cdf5 WatchSource:0}: Error finding container a3be40652bd724f348718337c599e3c5c5d400adabe0582742bbbec2c098cdf5: Status 404 returned error can't find the container with id a3be40652bd724f348718337c599e3c5c5d400adabe0582742bbbec2c098cdf5
Jul 25 01:30:00.308903 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:00.308881    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 25 01:30:01.279365 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:01.279309    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749050-pqr5s" event={"ID":"3e0cd0ad-e1ed-4788-b244-16dface55598","Type":"ContainerStarted","Data":"a3be40652bd724f348718337c599e3c5c5d400adabe0582742bbbec2c098cdf5"}
Jul 25 01:30:06.305069 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:06.305038    2572 generic.go:356] "Generic (PLEG): container finished" podID="3e0cd0ad-e1ed-4788-b244-16dface55598" containerID="c6f0e8ff741cd499b0f33ec2798e512f60da9ff182d7a6c357203b9d0d0e3039" exitCode=0
Jul 25 01:30:06.305494 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:06.305088    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749050-pqr5s" event={"ID":"3e0cd0ad-e1ed-4788-b244-16dface55598","Type":"ContainerDied","Data":"c6f0e8ff741cd499b0f33ec2798e512f60da9ff182d7a6c357203b9d0d0e3039"}
Jul 25 01:30:07.440679 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:07.440655    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749050-pqr5s"
Jul 25 01:30:08.315295 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:08.315257    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749050-pqr5s" event={"ID":"3e0cd0ad-e1ed-4788-b244-16dface55598","Type":"ContainerDied","Data":"a3be40652bd724f348718337c599e3c5c5d400adabe0582742bbbec2c098cdf5"}
Jul 25 01:30:08.315295 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:08.315280    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749050-pqr5s"
Jul 25 01:30:08.315527 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:30:08.315315    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3be40652bd724f348718337c599e3c5c5d400adabe0582742bbbec2c098cdf5"
Jul 25 01:37:00.195912 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.195879    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"]
Jul 25 01:37:00.198488 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.196224    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e0cd0ad-e1ed-4788-b244-16dface55598" containerName="cleanup"
Jul 25 01:37:00.198488 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.196236    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e0cd0ad-e1ed-4788-b244-16dface55598" containerName="cleanup"
Jul 25 01:37:00.198488 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.196304    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e0cd0ad-e1ed-4788-b244-16dface55598" containerName="cleanup"
Jul 25 01:37:00.199307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.199290    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.201562 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.201533    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c453b633-kserve-self-signed-certs\""
Jul 25 01:37:00.216173 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.216145    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"]
Jul 25 01:37:00.281182 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.281148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-dshm\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.281385 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.281190    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-model-cache\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.281385 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.281257    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-home\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.281385 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.281301    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-tmp-dir\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.281385 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.281354    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cc7d93-4ff7-40ea-9517-d999105c68d8-tls-certs\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.281385 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.281376    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k4z4\" (UniqueName: \"kubernetes.io/projected/76cc7d93-4ff7-40ea-9517-d999105c68d8-kube-api-access-9k4z4\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.382517 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.382485    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-dshm\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.382708 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.382536    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-model-cache\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.382708 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.382583    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-home\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.382708 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.382617    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-tmp-dir\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.382708 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.382654    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cc7d93-4ff7-40ea-9517-d999105c68d8-tls-certs\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.382708 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.382681    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k4z4\" (UniqueName: \"kubernetes.io/projected/76cc7d93-4ff7-40ea-9517-d999105c68d8-kube-api-access-9k4z4\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.382971 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.382949    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-model-cache\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.383041 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.383018    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-home\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.383086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.383039    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-tmp-dir\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.384827 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.384797    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-dshm\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.385074 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.385057    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cc7d93-4ff7-40ea-9517-d999105c68d8-tls-certs\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.390134 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.390114    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k4z4\" (UniqueName: \"kubernetes.io/projected/76cc7d93-4ff7-40ea-9517-d999105c68d8-kube-api-access-9k4z4\") pod \"e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") " pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.507754 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.507658    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:00.635961 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:00.635937    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"]
Jul 25 01:37:00.637971 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:37:00.637938    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76cc7d93_4ff7_40ea_9517_d999105c68d8.slice/crio-97108816ba218e88c00c1704b06629dc9dabe56cb30963e00b15fd0ca2f0c0ac WatchSource:0}: Error finding container 97108816ba218e88c00c1704b06629dc9dabe56cb30963e00b15fd0ca2f0c0ac: Status 404 returned error can't find the container with id 97108816ba218e88c00c1704b06629dc9dabe56cb30963e00b15fd0ca2f0c0ac
Jul 25 01:37:01.290638 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:01.290596    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52" event={"ID":"76cc7d93-4ff7-40ea-9517-d999105c68d8","Type":"ContainerStarted","Data":"ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2"}
Jul 25 01:37:01.290638 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:01.290643    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52" event={"ID":"76cc7d93-4ff7-40ea-9517-d999105c68d8","Type":"ContainerStarted","Data":"97108816ba218e88c00c1704b06629dc9dabe56cb30963e00b15fd0ca2f0c0ac"}
Jul 25 01:37:01.307312 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:01.307261    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52" podStartSLOduration=1.307245491 podStartE2EDuration="1.307245491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:37:01.3052285 +0000 UTC m=+1505.415037785" watchObservedRunningTime="2026-07-25 01:37:01.307245491 +0000 UTC m=+1505.417054775"
Jul 25 01:37:10.508698 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:10.508585    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:10.508698 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:10.508644    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:10.521163 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:10.521135    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:11.349850 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:11.349813    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:13.890050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:13.890003    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"]
Jul 25 01:37:13.893643 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:13.893626    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:13.895749 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:13.895728    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c453b633-kserve-self-signed-certs\""
Jul 25 01:37:13.900748 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:13.900722    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"]
Jul 25 01:37:13.999439 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:13.999403    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9170371-443f-4d17-945d-99a439b23e10-tls-certs\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:13.999439 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:13.999446    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-dshm\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:13.999646 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:13.999538    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-tmp-dir\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:13.999646 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:13.999568    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-model-cache\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:13.999646 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:13.999596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-home\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:13.999646 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:13.999637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-625dn\" (UniqueName: \"kubernetes.io/projected/a9170371-443f-4d17-945d-99a439b23e10-kube-api-access-625dn\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.100718 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.100679    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-tmp-dir\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.100718 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.100717    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-model-cache\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.100917 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.100738    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-home\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.100917 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.100865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-625dn\" (UniqueName: \"kubernetes.io/projected/a9170371-443f-4d17-945d-99a439b23e10-kube-api-access-625dn\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.101095 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.100936    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9170371-443f-4d17-945d-99a439b23e10-tls-certs\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.101095 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.100971    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-dshm\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.101196 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.101165    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-home\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.101249 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.101213    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-tmp-dir\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.101249 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.101231    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-model-cache\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.103175 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.103153    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-dshm\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.103550 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.103529    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9170371-443f-4d17-945d-99a439b23e10-tls-certs\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.107756 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.107734    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-625dn\" (UniqueName: \"kubernetes.io/projected/a9170371-443f-4d17-945d-99a439b23e10-kube-api-access-625dn\") pod \"e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") " pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.204087 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.204044    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:14.333234 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.333207    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"]
Jul 25 01:37:14.334421 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:37:14.334388    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9170371_443f_4d17_945d_99a439b23e10.slice/crio-ab63ee0e849f45965c79e1f1a005db633480b76ea77c027336c4bd4e7b811132 WatchSource:0}: Error finding container ab63ee0e849f45965c79e1f1a005db633480b76ea77c027336c4bd4e7b811132: Status 404 returned error can't find the container with id ab63ee0e849f45965c79e1f1a005db633480b76ea77c027336c4bd4e7b811132
Jul 25 01:37:14.354562 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:14.354523    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq" event={"ID":"a9170371-443f-4d17-945d-99a439b23e10","Type":"ContainerStarted","Data":"ab63ee0e849f45965c79e1f1a005db633480b76ea77c027336c4bd4e7b811132"}
Jul 25 01:37:15.360801 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:15.360767    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq" event={"ID":"a9170371-443f-4d17-945d-99a439b23e10","Type":"ContainerStarted","Data":"b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a"}
Jul 25 01:37:15.379867 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:15.379737    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq" podStartSLOduration=2.379714405 podStartE2EDuration="2.379714405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:37:15.375932997 +0000 UTC m=+1519.485742278" watchObservedRunningTime="2026-07-25 01:37:15.379714405 +0000 UTC m=+1519.489523691"
Jul 25 01:37:24.205234 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:24.205145    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:24.205234 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:24.205197    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:24.217861 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:24.217833    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:24.414084 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:24.414052    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:36.591108 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:36.591074    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"]
Jul 25 01:37:37.480562 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:37.480530    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"]
Jul 25 01:37:51.610263 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.610217    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52" podUID="76cc7d93-4ff7-40ea-9517-d999105c68d8" containerName="main" containerID="cri-o://ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2" gracePeriod=45
Jul 25 01:37:51.855508 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.855483    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:51.936211 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.936178    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cc7d93-4ff7-40ea-9517-d999105c68d8-tls-certs\") pod \"76cc7d93-4ff7-40ea-9517-d999105c68d8\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") "
Jul 25 01:37:51.936403 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.936232    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-dshm\") pod \"76cc7d93-4ff7-40ea-9517-d999105c68d8\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") "
Jul 25 01:37:51.936403 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.936251    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-tmp-dir\") pod \"76cc7d93-4ff7-40ea-9517-d999105c68d8\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") "
Jul 25 01:37:51.936403 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.936275    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-home\") pod \"76cc7d93-4ff7-40ea-9517-d999105c68d8\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") "
Jul 25 01:37:51.936403 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.936301    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-model-cache\") pod \"76cc7d93-4ff7-40ea-9517-d999105c68d8\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") "
Jul 25 01:37:51.936403 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.936355    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k4z4\" (UniqueName: \"kubernetes.io/projected/76cc7d93-4ff7-40ea-9517-d999105c68d8-kube-api-access-9k4z4\") pod \"76cc7d93-4ff7-40ea-9517-d999105c68d8\" (UID: \"76cc7d93-4ff7-40ea-9517-d999105c68d8\") "
Jul 25 01:37:51.936670 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.936539    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76cc7d93-4ff7-40ea-9517-d999105c68d8" (UID: "76cc7d93-4ff7-40ea-9517-d999105c68d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:37:51.936670 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.936581    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-home" (OuterVolumeSpecName: "home") pod "76cc7d93-4ff7-40ea-9517-d999105c68d8" (UID: "76cc7d93-4ff7-40ea-9517-d999105c68d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:37:51.936670 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.936599    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "76cc7d93-4ff7-40ea-9517-d999105c68d8" (UID: "76cc7d93-4ff7-40ea-9517-d999105c68d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:37:51.938521 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.938487    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cc7d93-4ff7-40ea-9517-d999105c68d8-kube-api-access-9k4z4" (OuterVolumeSpecName: "kube-api-access-9k4z4") pod "76cc7d93-4ff7-40ea-9517-d999105c68d8" (UID: "76cc7d93-4ff7-40ea-9517-d999105c68d8"). InnerVolumeSpecName "kube-api-access-9k4z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:37:51.938617 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.938528    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76cc7d93-4ff7-40ea-9517-d999105c68d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76cc7d93-4ff7-40ea-9517-d999105c68d8" (UID: "76cc7d93-4ff7-40ea-9517-d999105c68d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:37:51.938617 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:51.938557    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-dshm" (OuterVolumeSpecName: "dshm") pod "76cc7d93-4ff7-40ea-9517-d999105c68d8" (UID: "76cc7d93-4ff7-40ea-9517-d999105c68d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:37:52.037711 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.037676    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:52.037711 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.037704    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:52.037711 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.037712    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:52.037711 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.037720    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76cc7d93-4ff7-40ea-9517-d999105c68d8-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:52.038237 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.037731    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k4z4\" (UniqueName: \"kubernetes.io/projected/76cc7d93-4ff7-40ea-9517-d999105c68d8-kube-api-access-9k4z4\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:52.038237 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.037739    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76cc7d93-4ff7-40ea-9517-d999105c68d8-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:52.499307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.499266    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq" podUID="a9170371-443f-4d17-945d-99a439b23e10" containerName="main" containerID="cri-o://b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a" gracePeriod=45
Jul 25 01:37:52.553718 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.553679    2572 generic.go:356] "Generic (PLEG): container finished" podID="76cc7d93-4ff7-40ea-9517-d999105c68d8" containerID="ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2" exitCode=0
Jul 25 01:37:52.553877 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.553729    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"
Jul 25 01:37:52.553877 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.553753    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52" event={"ID":"76cc7d93-4ff7-40ea-9517-d999105c68d8","Type":"ContainerDied","Data":"ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2"}
Jul 25 01:37:52.553877 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.553792    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52" event={"ID":"76cc7d93-4ff7-40ea-9517-d999105c68d8","Type":"ContainerDied","Data":"97108816ba218e88c00c1704b06629dc9dabe56cb30963e00b15fd0ca2f0c0ac"}
Jul 25 01:37:52.553877 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.553811    2572 scope.go:117] "RemoveContainer" containerID="ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2"
Jul 25 01:37:52.571212 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.571185    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"]
Jul 25 01:37:52.575135 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.575111    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c453b633-kserve-6667c485ff-tzz52"]
Jul 25 01:37:52.610113 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.610094    2572 scope.go:117] "RemoveContainer" containerID="ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2"
Jul 25 01:37:52.610461 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:37:52.610434    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2\": container with ID starting with ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2 not found: ID does not exist" containerID="ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2"
Jul 25 01:37:52.610751 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.610471    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2"} err="failed to get container status \"ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2\": rpc error: code = NotFound desc = could not find container \"ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2\": container with ID starting with ff1c3691760fe02339f3d01bb3dcf67c99900a36fadd37f200a7120f8d7d2ce2 not found: ID does not exist"
Jul 25 01:37:52.747644 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.747617    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:52.845472 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.845422    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-625dn\" (UniqueName: \"kubernetes.io/projected/a9170371-443f-4d17-945d-99a439b23e10-kube-api-access-625dn\") pod \"a9170371-443f-4d17-945d-99a439b23e10\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") "
Jul 25 01:37:52.845472 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.845482    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-home\") pod \"a9170371-443f-4d17-945d-99a439b23e10\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") "
Jul 25 01:37:52.845692 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.845520    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9170371-443f-4d17-945d-99a439b23e10-tls-certs\") pod \"a9170371-443f-4d17-945d-99a439b23e10\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") "
Jul 25 01:37:52.845692 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.845631    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-tmp-dir\") pod \"a9170371-443f-4d17-945d-99a439b23e10\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") "
Jul 25 01:37:52.845692 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.845672    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-dshm\") pod \"a9170371-443f-4d17-945d-99a439b23e10\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") "
Jul 25 01:37:52.845923 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.845703    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-model-cache\") pod \"a9170371-443f-4d17-945d-99a439b23e10\" (UID: \"a9170371-443f-4d17-945d-99a439b23e10\") "
Jul 25 01:37:52.845923 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.845793    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-home" (OuterVolumeSpecName: "home") pod "a9170371-443f-4d17-945d-99a439b23e10" (UID: "a9170371-443f-4d17-945d-99a439b23e10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:37:52.845923 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.845893    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9170371-443f-4d17-945d-99a439b23e10" (UID: "a9170371-443f-4d17-945d-99a439b23e10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:37:52.846079 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.846025    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:52.846079 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.846040    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9170371-443f-4d17-945d-99a439b23e10" (UID: "a9170371-443f-4d17-945d-99a439b23e10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:37:52.846079 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.846066    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:52.847690 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.847667    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9170371-443f-4d17-945d-99a439b23e10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9170371-443f-4d17-945d-99a439b23e10" (UID: "a9170371-443f-4d17-945d-99a439b23e10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:37:52.847784 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.847693    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-dshm" (OuterVolumeSpecName: "dshm") pod "a9170371-443f-4d17-945d-99a439b23e10" (UID: "a9170371-443f-4d17-945d-99a439b23e10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:37:52.847784 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.847708    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9170371-443f-4d17-945d-99a439b23e10-kube-api-access-625dn" (OuterVolumeSpecName: "kube-api-access-625dn") pod "a9170371-443f-4d17-945d-99a439b23e10" (UID: "a9170371-443f-4d17-945d-99a439b23e10"). InnerVolumeSpecName "kube-api-access-625dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:37:52.947023 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.946974    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9170371-443f-4d17-945d-99a439b23e10-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:52.947023 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.947015    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:52.947023 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.947025    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9170371-443f-4d17-945d-99a439b23e10-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:52.947023 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:52.947033    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-625dn\" (UniqueName: \"kubernetes.io/projected/a9170371-443f-4d17-945d-99a439b23e10-kube-api-access-625dn\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:37:53.560777 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:53.560746    2572 generic.go:356] "Generic (PLEG): container finished" podID="a9170371-443f-4d17-945d-99a439b23e10" containerID="b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a" exitCode=0
Jul 25 01:37:53.560957 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:53.560783    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"
Jul 25 01:37:53.560957 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:53.560804    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq" event={"ID":"a9170371-443f-4d17-945d-99a439b23e10","Type":"ContainerDied","Data":"b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a"}
Jul 25 01:37:53.560957 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:53.560839    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq" event={"ID":"a9170371-443f-4d17-945d-99a439b23e10","Type":"ContainerDied","Data":"ab63ee0e849f45965c79e1f1a005db633480b76ea77c027336c4bd4e7b811132"}
Jul 25 01:37:53.560957 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:53.560858    2572 scope.go:117] "RemoveContainer" containerID="b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a"
Jul 25 01:37:53.570241 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:53.570226    2572 scope.go:117] "RemoveContainer" containerID="b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a"
Jul 25 01:37:53.570509 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:37:53.570488    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a\": container with ID starting with b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a not found: ID does not exist" containerID="b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a"
Jul 25 01:37:53.570569 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:53.570517    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a"} err="failed to get container status \"b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a\": rpc error: code = NotFound desc = could not find container \"b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a\": container with ID starting with b816f773cf6e110e612496f3779625e0f4dd7f24c24bd6122382ca8b535fe49a not found: ID does not exist"
Jul 25 01:37:53.581771 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:53.581741    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"]
Jul 25 01:37:53.585590 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:53.585568    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c453b633-kserve-7974cdc-rs5fq"]
Jul 25 01:37:54.352695 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:54.352664    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76cc7d93-4ff7-40ea-9517-d999105c68d8" path="/var/lib/kubelet/pods/76cc7d93-4ff7-40ea-9517-d999105c68d8/volumes"
Jul 25 01:37:54.353039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:37:54.353027    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9170371-443f-4d17-945d-99a439b23e10" path="/var/lib/kubelet/pods/a9170371-443f-4d17-945d-99a439b23e10/volumes"
Jul 25 01:41:24.516316 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.516277    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"]
Jul 25 01:41:24.516849 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.516739    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9170371-443f-4d17-945d-99a439b23e10" containerName="main"
Jul 25 01:41:24.516849 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.516754    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9170371-443f-4d17-945d-99a439b23e10" containerName="main"
Jul 25 01:41:24.516849 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.516774    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76cc7d93-4ff7-40ea-9517-d999105c68d8" containerName="main"
Jul 25 01:41:24.516849 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.516783    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76cc7d93-4ff7-40ea-9517-d999105c68d8" containerName="main"
Jul 25 01:41:24.517051 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.516861    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9170371-443f-4d17-945d-99a439b23e10" containerName="main"
Jul 25 01:41:24.517051 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.516875    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76cc7d93-4ff7-40ea-9517-d999105c68d8" containerName="main"
Jul 25 01:41:24.520051 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.520030    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:24.522596 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.522573    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-02d4b958-serving-cert\""
Jul 25 01:41:24.522749 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.522729    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhv77\""
Jul 25 01:41:24.528972 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.528939    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"]
Jul 25 01:41:24.636027 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.635984    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124606b5-97ac-494f-9284-e4ee8d368eb4-maas-api-tls\") pod \"maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:24.636228 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.636043    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/124606b5-97ac-494f-9284-e4ee8d368eb4-openshift-service-ca\") pod \"maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:24.636228 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.636149    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9w5k\" (UniqueName: \"kubernetes.io/projected/124606b5-97ac-494f-9284-e4ee8d368eb4-kube-api-access-v9w5k\") pod \"maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:24.736932 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.736893    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9w5k\" (UniqueName: \"kubernetes.io/projected/124606b5-97ac-494f-9284-e4ee8d368eb4-kube-api-access-v9w5k\") pod \"maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:24.737121 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.736939    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124606b5-97ac-494f-9284-e4ee8d368eb4-maas-api-tls\") pod \"maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:24.737121 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.736978    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/124606b5-97ac-494f-9284-e4ee8d368eb4-openshift-service-ca\") pod \"maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:24.737121 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:41:24.737102    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-serving-cert: secret "maas-api-e2e-ait-02d4b958-serving-cert" not found
Jul 25 01:41:24.737294 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:41:24.737178    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/124606b5-97ac-494f-9284-e4ee8d368eb4-maas-api-tls podName:124606b5-97ac-494f-9284-e4ee8d368eb4 nodeName:}" failed. No retries permitted until 2026-07-25 01:41:25.237161001 +0000 UTC m=+1769.346970282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/124606b5-97ac-494f-9284-e4ee8d368eb4-maas-api-tls") pod "maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2" (UID: "124606b5-97ac-494f-9284-e4ee8d368eb4") : secret "maas-api-e2e-ait-02d4b958-serving-cert" not found
Jul 25 01:41:24.737705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.737683    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/124606b5-97ac-494f-9284-e4ee8d368eb4-openshift-service-ca\") pod \"maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:24.750011 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:24.749978    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9w5k\" (UniqueName: \"kubernetes.io/projected/124606b5-97ac-494f-9284-e4ee8d368eb4-kube-api-access-v9w5k\") pod \"maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:25.240845 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:25.240813    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124606b5-97ac-494f-9284-e4ee8d368eb4-maas-api-tls\") pod \"maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:25.243311 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:25.243279    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124606b5-97ac-494f-9284-e4ee8d368eb4-maas-api-tls\") pod \"maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:25.431549 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:25.431510    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:25.562226 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:25.562197    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"]
Jul 25 01:41:25.564026 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:41:25.564002    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124606b5_97ac_494f_9284_e4ee8d368eb4.slice/crio-dd6ff68e2ada4244346f3d3e305425ea393352f16494aa209bd8a176a98fcbfd WatchSource:0}: Error finding container dd6ff68e2ada4244346f3d3e305425ea393352f16494aa209bd8a176a98fcbfd: Status 404 returned error can't find the container with id dd6ff68e2ada4244346f3d3e305425ea393352f16494aa209bd8a176a98fcbfd
Jul 25 01:41:25.565506 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:25.565488    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 25 01:41:25.586424 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:25.586394    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2" event={"ID":"124606b5-97ac-494f-9284-e4ee8d368eb4","Type":"ContainerStarted","Data":"dd6ff68e2ada4244346f3d3e305425ea393352f16494aa209bd8a176a98fcbfd"}
Jul 25 01:41:26.592783 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:26.592750    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2" event={"ID":"124606b5-97ac-494f-9284-e4ee8d368eb4","Type":"ContainerStarted","Data":"3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb"}
Jul 25 01:41:26.593169 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:26.592824    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:26.611928 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:26.611878    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2" podStartSLOduration=2.181647079 podStartE2EDuration="2.611863079s" totalImagesPullingTime="430.216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:41:26.609269377 +0000 UTC m=+1770.719078662" watchObservedRunningTime="2026-07-25 01:41:26.611863079 +0000 UTC m=+1770.721672417"
Jul 25 01:41:32.605147 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:32.605116    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:41:50.076767 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:50.076691    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"]
Jul 25 01:41:50.080093 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:50.080070    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"
Jul 25 01:41:50.082393 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:50.082359    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 01:41:50.086487 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:50.086458    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"]
Jul 25 01:41:50.254596 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:50.254556    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ff2bc49-721e-417b-9b53-d4a71596b957-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02d4b958-r9c59\" (UID: \"9ff2bc49-721e-417b-9b53-d4a71596b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"
Jul 25 01:41:50.355109 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:50.355036    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ff2bc49-721e-417b-9b53-d4a71596b957-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02d4b958-r9c59\" (UID: \"9ff2bc49-721e-417b-9b53-d4a71596b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"
Jul 25 01:41:50.355683 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:50.355662    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ff2bc49-721e-417b-9b53-d4a71596b957-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-02d4b958-r9c59\" (UID: \"9ff2bc49-721e-417b-9b53-d4a71596b957\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"
Jul 25 01:41:50.392021 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:50.391991    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"
Jul 25 01:41:50.520125 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:50.520097    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"]
Jul 25 01:41:50.522011 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:41:50.521983    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff2bc49_721e_417b_9b53_d4a71596b957.slice/crio-f6460b2dd1fa3dac164de52fcf1daf2b3b3329370c3ca6a53b8c2b96def02dcf WatchSource:0}: Error finding container f6460b2dd1fa3dac164de52fcf1daf2b3b3329370c3ca6a53b8c2b96def02dcf: Status 404 returned error can't find the container with id f6460b2dd1fa3dac164de52fcf1daf2b3b3329370c3ca6a53b8c2b96def02dcf
Jul 25 01:41:50.710620 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:50.710587    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59" event={"ID":"9ff2bc49-721e-417b-9b53-d4a71596b957","Type":"ContainerStarted","Data":"f6460b2dd1fa3dac164de52fcf1daf2b3b3329370c3ca6a53b8c2b96def02dcf"}
Jul 25 01:41:51.716639 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:51.716601    2572 generic.go:356] "Generic (PLEG): container finished" podID="9ff2bc49-721e-417b-9b53-d4a71596b957" containerID="4c1488dcf4a5ba759b2bee4a45b231b787c200e3229ea5a49c1fb169de53ee79" exitCode=0
Jul 25 01:41:51.717043 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:51.716649    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59" event={"ID":"9ff2bc49-721e-417b-9b53-d4a71596b957","Type":"ContainerDied","Data":"4c1488dcf4a5ba759b2bee4a45b231b787c200e3229ea5a49c1fb169de53ee79"}
Jul 25 01:41:52.848846 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:52.848820    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"
Jul 25 01:41:52.878573 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:52.878545    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ff2bc49-721e-417b-9b53-d4a71596b957-maas-api-service-ca\") pod \"9ff2bc49-721e-417b-9b53-d4a71596b957\" (UID: \"9ff2bc49-721e-417b-9b53-d4a71596b957\") "
Jul 25 01:41:52.878891 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:52.878870    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff2bc49-721e-417b-9b53-d4a71596b957-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ff2bc49-721e-417b-9b53-d4a71596b957" (UID: "9ff2bc49-721e-417b-9b53-d4a71596b957"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:41:52.979185 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:52.979101    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ff2bc49-721e-417b-9b53-d4a71596b957-maas-api-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:41:53.726970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:53.726927    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59" event={"ID":"9ff2bc49-721e-417b-9b53-d4a71596b957","Type":"ContainerDied","Data":"f6460b2dd1fa3dac164de52fcf1daf2b3b3329370c3ca6a53b8c2b96def02dcf"}
Jul 25 01:41:53.726970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:53.726957    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"
Jul 25 01:41:53.727173 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:41:53.726992    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6460b2dd1fa3dac164de52fcf1daf2b3b3329370c3ca6a53b8c2b96def02dcf"
Jul 25 01:42:00.661037 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:00.661000    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"]
Jul 25 01:42:00.663466 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:00.661238    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2" podUID="124606b5-97ac-494f-9284-e4ee8d368eb4" containerName="maas-api" containerID="cri-o://3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb" gracePeriod=30
Jul 25 01:42:00.697642 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:42:00.697617    2572 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124606b5_97ac_494f_9284_e4ee8d368eb4.slice/crio-3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb.scope\": RecentStats: unable to find data in memory cache]"
Jul 25 01:42:00.900615 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:00.900590    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:42:00.943998 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:00.943912    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9w5k\" (UniqueName: \"kubernetes.io/projected/124606b5-97ac-494f-9284-e4ee8d368eb4-kube-api-access-v9w5k\") pod \"124606b5-97ac-494f-9284-e4ee8d368eb4\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") "
Jul 25 01:42:00.943998 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:00.943971    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/124606b5-97ac-494f-9284-e4ee8d368eb4-openshift-service-ca\") pod \"124606b5-97ac-494f-9284-e4ee8d368eb4\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") "
Jul 25 01:42:00.943998 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:00.943992    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124606b5-97ac-494f-9284-e4ee8d368eb4-maas-api-tls\") pod \"124606b5-97ac-494f-9284-e4ee8d368eb4\" (UID: \"124606b5-97ac-494f-9284-e4ee8d368eb4\") "
Jul 25 01:42:00.944348 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:00.944305    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124606b5-97ac-494f-9284-e4ee8d368eb4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "124606b5-97ac-494f-9284-e4ee8d368eb4" (UID: "124606b5-97ac-494f-9284-e4ee8d368eb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:42:00.946230 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:00.946205    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124606b5-97ac-494f-9284-e4ee8d368eb4-kube-api-access-v9w5k" (OuterVolumeSpecName: "kube-api-access-v9w5k") pod "124606b5-97ac-494f-9284-e4ee8d368eb4" (UID: "124606b5-97ac-494f-9284-e4ee8d368eb4"). InnerVolumeSpecName "kube-api-access-v9w5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:42:00.946230 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:00.946217    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124606b5-97ac-494f-9284-e4ee8d368eb4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "124606b5-97ac-494f-9284-e4ee8d368eb4" (UID: "124606b5-97ac-494f-9284-e4ee8d368eb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:42:01.044900 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.044862    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/124606b5-97ac-494f-9284-e4ee8d368eb4-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:01.044900 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.044893    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/124606b5-97ac-494f-9284-e4ee8d368eb4-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:01.044900 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.044905    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9w5k\" (UniqueName: \"kubernetes.io/projected/124606b5-97ac-494f-9284-e4ee8d368eb4-kube-api-access-v9w5k\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:01.768263 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.768229    2572 generic.go:356] "Generic (PLEG): container finished" podID="124606b5-97ac-494f-9284-e4ee8d368eb4" containerID="3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb" exitCode=0
Jul 25 01:42:01.768735 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.768269    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2" event={"ID":"124606b5-97ac-494f-9284-e4ee8d368eb4","Type":"ContainerDied","Data":"3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb"}
Jul 25 01:42:01.768735 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.768274    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"
Jul 25 01:42:01.768735 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.768310    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2" event={"ID":"124606b5-97ac-494f-9284-e4ee8d368eb4","Type":"ContainerDied","Data":"dd6ff68e2ada4244346f3d3e305425ea393352f16494aa209bd8a176a98fcbfd"}
Jul 25 01:42:01.768735 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.768369    2572 scope.go:117] "RemoveContainer" containerID="3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb"
Jul 25 01:42:01.777922 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.777903    2572 scope.go:117] "RemoveContainer" containerID="3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb"
Jul 25 01:42:01.778175 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:42:01.778157    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb\": container with ID starting with 3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb not found: ID does not exist" containerID="3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb"
Jul 25 01:42:01.778221 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.778186    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb"} err="failed to get container status \"3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb\": rpc error: code = NotFound desc = could not find container \"3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb\": container with ID starting with 3f222b45c5a08e2482d9c3a95906ca2da8e7cac03cb3111199c64b3021be36fb not found: ID does not exist"
Jul 25 01:42:01.789574 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.789553    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"]
Jul 25 01:42:01.791500 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:01.791479    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-02d4b958-6f4b87cd76-zdcb2"]
Jul 25 01:42:02.352576 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:02.352540    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="124606b5-97ac-494f-9284-e4ee8d368eb4" path="/var/lib/kubelet/pods/124606b5-97ac-494f-9284-e4ee8d368eb4/volumes"
Jul 25 01:42:05.145345 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:05.145283    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"]
Jul 25 01:42:05.147350 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:05.147313    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-02d4b958-r9c59"]
Jul 25 01:42:06.334973 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.334940    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"]
Jul 25 01:42:06.335348 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.335316    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124606b5-97ac-494f-9284-e4ee8d368eb4" containerName="maas-api"
Jul 25 01:42:06.335348 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.335341    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="124606b5-97ac-494f-9284-e4ee8d368eb4" containerName="maas-api"
Jul 25 01:42:06.335429 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.335356    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ff2bc49-721e-417b-9b53-d4a71596b957" containerName="revoke-keys"
Jul 25 01:42:06.335429 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.335361    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ff2bc49-721e-417b-9b53-d4a71596b957" containerName="revoke-keys"
Jul 25 01:42:06.335429 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.335413    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ff2bc49-721e-417b-9b53-d4a71596b957" containerName="revoke-keys"
Jul 25 01:42:06.335429 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.335426    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="124606b5-97ac-494f-9284-e4ee8d368eb4" containerName="maas-api"
Jul 25 01:42:06.340093 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.340067    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.342565 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.342544    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 25 01:42:06.342688 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.342544    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-842d4926-gw-openshift-default-dockercfg-sqzdj\""
Jul 25 01:42:06.354428 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.354402    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ff2bc49-721e-417b-9b53-d4a71596b957" path="/var/lib/kubelet/pods/9ff2bc49-721e-417b-9b53-d4a71596b957/volumes"
Jul 25 01:42:06.354714 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.354697    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"]
Jul 25 01:42:06.387596 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.387555    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qx9\" (UniqueName: \"kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-kube-api-access-k6qx9\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.387596 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.387608    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-envoy\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.387826 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.387648    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18850452-e46a-4667-8846-a95fa5da18eb-istio-podinfo\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.387826 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.387741    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-certs\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.387826 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.387778    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-data\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.387826 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.387803    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-istio-token\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.388022 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.387830    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18850452-e46a-4667-8846-a95fa5da18eb-istiod-ca-cert\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.388022 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.387903    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-socket\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.388022 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.387937    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-credential-socket\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489026 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.488989    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-data\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489026 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489035    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-istio-token\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489245 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18850452-e46a-4667-8846-a95fa5da18eb-istiod-ca-cert\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489245 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489240    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-socket\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489365 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489258    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-credential-socket\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489365 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489302    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qx9\" (UniqueName: \"kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-kube-api-access-k6qx9\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489365 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489341    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-envoy\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489529 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489391    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-data\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489529 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489505    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18850452-e46a-4667-8846-a95fa5da18eb-istio-podinfo\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489591    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-socket\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489618    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-certs\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489750 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489706    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-credential-socket\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489921 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489900    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-certs\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.489969 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.489950    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18850452-e46a-4667-8846-a95fa5da18eb-istiod-ca-cert\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.491558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.491527    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-envoy\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.491982 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.491961    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18850452-e46a-4667-8846-a95fa5da18eb-istio-podinfo\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.497878 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.497851    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qx9\" (UniqueName: \"kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-kube-api-access-k6qx9\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.498045 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.498025    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-istio-token\") pod \"e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") " pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.651548 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.651520    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:06.785598 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:06.785566    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"]
Jul 25 01:42:06.787565 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:42:06.787538    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18850452_e46a_4667_8846_a95fa5da18eb.slice/crio-a1a8cc3dfad9b403d74e0f9ab1662de80fddb4da2bb10c133639d25864e59024 WatchSource:0}: Error finding container a1a8cc3dfad9b403d74e0f9ab1662de80fddb4da2bb10c133639d25864e59024: Status 404 returned error can't find the container with id a1a8cc3dfad9b403d74e0f9ab1662de80fddb4da2bb10c133639d25864e59024
Jul 25 01:42:07.799672 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:07.799633    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" event={"ID":"18850452-e46a-4667-8846-a95fa5da18eb","Type":"ContainerStarted","Data":"a1a8cc3dfad9b403d74e0f9ab1662de80fddb4da2bb10c133639d25864e59024"}
Jul 25 01:42:10.376942 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:10.376905    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:42:10.377184 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:10.376977    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:42:10.377184 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:10.377003    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:42:10.816403 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:10.816367    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" event={"ID":"18850452-e46a-4667-8846-a95fa5da18eb","Type":"ContainerStarted","Data":"db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8"}
Jul 25 01:42:10.834421 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:10.834354    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" podStartSLOduration=1.247503775 podStartE2EDuration="4.834308838s" totalImagesPullingTime="3.586805063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:42:10.833884736 +0000 UTC m=+1814.943694021" watchObservedRunningTime="2026-07-25 01:42:10.834308838 +0000 UTC m=+1814.944118135"
Jul 25 01:42:11.652468 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:11.652430    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:11.657044 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:11.657019    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:11.821677 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:11.821646    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:11.822451 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:11.822430    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:12.359613 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.359581    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"]
Jul 25 01:42:12.363315 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.363293    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:12.365833 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.365810    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-842d4926-serving-cert\""
Jul 25 01:42:12.366146 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.366124    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhv77\""
Jul 25 01:42:12.375733 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.375706    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"]
Jul 25 01:42:12.443566 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.443534    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4200e2bc-b220-4237-b298-8bf1d719d454-openshift-service-ca\") pod \"maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:12.443709 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.443585    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4cvl\" (UniqueName: \"kubernetes.io/projected/4200e2bc-b220-4237-b298-8bf1d719d454-kube-api-access-n4cvl\") pod \"maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:12.443709 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.443674    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4200e2bc-b220-4237-b298-8bf1d719d454-maas-api-tls\") pod \"maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:12.544577 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.544535    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cvl\" (UniqueName: \"kubernetes.io/projected/4200e2bc-b220-4237-b298-8bf1d719d454-kube-api-access-n4cvl\") pod \"maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:12.544749 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.544606    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4200e2bc-b220-4237-b298-8bf1d719d454-maas-api-tls\") pod \"maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:12.544749 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.544645    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4200e2bc-b220-4237-b298-8bf1d719d454-openshift-service-ca\") pod \"maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:12.544829 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:42:12.544745    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-serving-cert: secret "maas-api-e2e-migrate-842d4926-serving-cert" not found
Jul 25 01:42:12.544864 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:42:12.544833    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4200e2bc-b220-4237-b298-8bf1d719d454-maas-api-tls podName:4200e2bc-b220-4237-b298-8bf1d719d454 nodeName:}" failed. No retries permitted until 2026-07-25 01:42:13.044811484 +0000 UTC m=+1817.154620760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4200e2bc-b220-4237-b298-8bf1d719d454-maas-api-tls") pod "maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp" (UID: "4200e2bc-b220-4237-b298-8bf1d719d454") : secret "maas-api-e2e-migrate-842d4926-serving-cert" not found
Jul 25 01:42:12.545213 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.545196    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4200e2bc-b220-4237-b298-8bf1d719d454-openshift-service-ca\") pod \"maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:12.554757 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:12.554730    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cvl\" (UniqueName: \"kubernetes.io/projected/4200e2bc-b220-4237-b298-8bf1d719d454-kube-api-access-n4cvl\") pod \"maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:13.050595 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:13.050553    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4200e2bc-b220-4237-b298-8bf1d719d454-maas-api-tls\") pod \"maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:13.052971 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:13.052945    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4200e2bc-b220-4237-b298-8bf1d719d454-maas-api-tls\") pod \"maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:13.276512 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:13.276474    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:13.406466 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:13.406439    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"]
Jul 25 01:42:13.408997 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:42:13.408966    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4200e2bc_b220_4237_b298_8bf1d719d454.slice/crio-627d94d333913bdacb04946f1fe037138486a2ede88dcbf5e12d60c06f085fd7 WatchSource:0}: Error finding container 627d94d333913bdacb04946f1fe037138486a2ede88dcbf5e12d60c06f085fd7: Status 404 returned error can't find the container with id 627d94d333913bdacb04946f1fe037138486a2ede88dcbf5e12d60c06f085fd7
Jul 25 01:42:13.831625 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:13.831598    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp" event={"ID":"4200e2bc-b220-4237-b298-8bf1d719d454","Type":"ContainerStarted","Data":"627d94d333913bdacb04946f1fe037138486a2ede88dcbf5e12d60c06f085fd7"}
Jul 25 01:42:14.839161 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:14.839127    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp" event={"ID":"4200e2bc-b220-4237-b298-8bf1d719d454","Type":"ContainerStarted","Data":"ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95"}
Jul 25 01:42:14.839566 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:14.839222    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:14.854834 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:14.854786    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp" podStartSLOduration=2.439485663 podStartE2EDuration="2.854771784s" totalImagesPullingTime="415.286121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:42:14.853621935 +0000 UTC m=+1818.963431222" watchObservedRunningTime="2026-07-25 01:42:14.854771784 +0000 UTC m=+1818.964581068"
Jul 25 01:42:20.860459 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:20.860425    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:23.905940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:23.905905    2572 generic.go:356] "Generic (PLEG): container finished" podID="18850452-e46a-4667-8846-a95fa5da18eb" containerID="db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8" exitCode=0
Jul 25 01:42:23.906403 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:23.905979    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" event={"ID":"18850452-e46a-4667-8846-a95fa5da18eb","Type":"ContainerDied","Data":"db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8"}
Jul 25 01:42:23.906403 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:23.906388    2572 scope.go:117] "RemoveContainer" containerID="db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8"
Jul 25 01:42:23.908356 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:23.908308    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:42:23.908477 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:23.908429    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:42:23.908543 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:23.908476    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:42:24.913009 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:24.912979    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" event={"ID":"18850452-e46a-4667-8846-a95fa5da18eb","Type":"ContainerStarted","Data":"b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf"}
Jul 25 01:42:25.653651 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:25.652514    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:25.653991 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:25.653962    2572 patch_prober.go:28] interesting pod/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 25 01:42:25.654188 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:25.654153    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" podUID="18850452-e46a-4667-8846-a95fa5da18eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 25 01:42:26.652649 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:26.652611    2572 patch_prober.go:28] interesting pod/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 25 01:42:26.653143 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:26.652715    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" podUID="18850452-e46a-4667-8846-a95fa5da18eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 25 01:42:26.822570 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:26.822544    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:27.652031 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:27.651988    2572 patch_prober.go:28] interesting pod/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 25 01:42:27.652411 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:27.652377    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" podUID="18850452-e46a-4667-8846-a95fa5da18eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 25 01:42:28.652171 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:28.652131    2572 patch_prober.go:28] interesting pod/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 25 01:42:28.652776 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:28.652741    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" podUID="18850452-e46a-4667-8846-a95fa5da18eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 25 01:42:29.656032 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:29.655999    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:29.657161 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:29.657138    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:37.323589 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:37.323548    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"]
Jul 25 01:42:37.332027 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:37.332001    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"
Jul 25 01:42:37.333851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:37.333827    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"]
Jul 25 01:42:37.334355 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:37.334315    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 01:42:37.473707 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:37.473664    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-842d4926-tqssj\" (UID: \"aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"
Jul 25 01:42:37.575082 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:37.574985    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-842d4926-tqssj\" (UID: \"aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"
Jul 25 01:42:37.575679 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:37.575652    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-842d4926-tqssj\" (UID: \"aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"
Jul 25 01:42:37.645084 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:37.645041    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"
Jul 25 01:42:37.775795 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:37.775765    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"]
Jul 25 01:42:37.778604 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:42:37.778572    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed2d0a6_79eb_45b3_bf0a_fb6ca7b4b9fd.slice/crio-e59fb80e2dc142d11594dd761a68386fa42882222ba95e35750d3b0cf8f7c394 WatchSource:0}: Error finding container e59fb80e2dc142d11594dd761a68386fa42882222ba95e35750d3b0cf8f7c394: Status 404 returned error can't find the container with id e59fb80e2dc142d11594dd761a68386fa42882222ba95e35750d3b0cf8f7c394
Jul 25 01:42:37.977095 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:37.977065    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj" event={"ID":"aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd","Type":"ContainerStarted","Data":"e59fb80e2dc142d11594dd761a68386fa42882222ba95e35750d3b0cf8f7c394"}
Jul 25 01:42:38.983374 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:38.983312    2572 generic.go:356] "Generic (PLEG): container finished" podID="aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd" containerID="de79f1ebd9231f87ec38af8b9992a70b86ff4e668ef0c3d2b9b24cf22a60b9b2" exitCode=0
Jul 25 01:42:38.983374 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:38.983351    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj" event={"ID":"aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd","Type":"ContainerDied","Data":"de79f1ebd9231f87ec38af8b9992a70b86ff4e668ef0c3d2b9b24cf22a60b9b2"}
Jul 25 01:42:40.113766 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:40.113734    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"
Jul 25 01:42:40.300516 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:40.300420    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd-maas-api-service-ca\") pod \"aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd\" (UID: \"aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd\") "
Jul 25 01:42:40.300781 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:40.300757    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd" (UID: "aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:42:40.401542 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:40.401503    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd-maas-api-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:40.994421 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:40.994391    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"
Jul 25 01:42:40.994588 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:40.994387    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj" event={"ID":"aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd","Type":"ContainerDied","Data":"e59fb80e2dc142d11594dd761a68386fa42882222ba95e35750d3b0cf8f7c394"}
Jul 25 01:42:40.994588 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:40.994525    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e59fb80e2dc142d11594dd761a68386fa42882222ba95e35750d3b0cf8f7c394"
Jul 25 01:42:47.718256 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:47.718218    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"]
Jul 25 01:42:47.718694 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:47.718513    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp" podUID="4200e2bc-b220-4237-b298-8bf1d719d454" containerName="maas-api" containerID="cri-o://ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95" gracePeriod=30
Jul 25 01:42:47.964162 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:47.964137    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:48.030504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.030400    2572 generic.go:356] "Generic (PLEG): container finished" podID="4200e2bc-b220-4237-b298-8bf1d719d454" containerID="ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95" exitCode=0
Jul 25 01:42:48.030504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.030450    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"
Jul 25 01:42:48.030504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.030485    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp" event={"ID":"4200e2bc-b220-4237-b298-8bf1d719d454","Type":"ContainerDied","Data":"ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95"}
Jul 25 01:42:48.030744 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.030542    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp" event={"ID":"4200e2bc-b220-4237-b298-8bf1d719d454","Type":"ContainerDied","Data":"627d94d333913bdacb04946f1fe037138486a2ede88dcbf5e12d60c06f085fd7"}
Jul 25 01:42:48.030744 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.030566    2572 scope.go:117] "RemoveContainer" containerID="ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95"
Jul 25 01:42:48.038734 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.038716    2572 scope.go:117] "RemoveContainer" containerID="ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95"
Jul 25 01:42:48.038990 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:42:48.038970    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95\": container with ID starting with ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95 not found: ID does not exist" containerID="ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95"
Jul 25 01:42:48.039045 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.038998    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95"} err="failed to get container status \"ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95\": rpc error: code = NotFound desc = could not find container \"ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95\": container with ID starting with ba06d9f489748ea8e1b75f65862902510a826704a7ee2b7f8beb2d222c99fc95 not found: ID does not exist"
Jul 25 01:42:48.065250 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.065225    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4200e2bc-b220-4237-b298-8bf1d719d454-openshift-service-ca\") pod \"4200e2bc-b220-4237-b298-8bf1d719d454\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") "
Jul 25 01:42:48.065401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.065288    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4cvl\" (UniqueName: \"kubernetes.io/projected/4200e2bc-b220-4237-b298-8bf1d719d454-kube-api-access-n4cvl\") pod \"4200e2bc-b220-4237-b298-8bf1d719d454\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") "
Jul 25 01:42:48.065401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.065339    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4200e2bc-b220-4237-b298-8bf1d719d454-maas-api-tls\") pod \"4200e2bc-b220-4237-b298-8bf1d719d454\" (UID: \"4200e2bc-b220-4237-b298-8bf1d719d454\") "
Jul 25 01:42:48.065640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.065606    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4200e2bc-b220-4237-b298-8bf1d719d454-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4200e2bc-b220-4237-b298-8bf1d719d454" (UID: "4200e2bc-b220-4237-b298-8bf1d719d454"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:42:48.067313 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.067280    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4200e2bc-b220-4237-b298-8bf1d719d454-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4200e2bc-b220-4237-b298-8bf1d719d454" (UID: "4200e2bc-b220-4237-b298-8bf1d719d454"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:42:48.067418 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.067342    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4200e2bc-b220-4237-b298-8bf1d719d454-kube-api-access-n4cvl" (OuterVolumeSpecName: "kube-api-access-n4cvl") pod "4200e2bc-b220-4237-b298-8bf1d719d454" (UID: "4200e2bc-b220-4237-b298-8bf1d719d454"). InnerVolumeSpecName "kube-api-access-n4cvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:42:48.166048 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.166016    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4200e2bc-b220-4237-b298-8bf1d719d454-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:48.166048 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.166045    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4cvl\" (UniqueName: \"kubernetes.io/projected/4200e2bc-b220-4237-b298-8bf1d719d454-kube-api-access-n4cvl\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:48.166048 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.166055    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4200e2bc-b220-4237-b298-8bf1d719d454-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:48.354527 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.354500    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"]
Jul 25 01:42:48.357907 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:48.357881    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-842d4926-59b86b7788-8hdzp"]
Jul 25 01:42:50.353275 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:50.353241    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4200e2bc-b220-4237-b298-8bf1d719d454" path="/var/lib/kubelet/pods/4200e2bc-b220-4237-b298-8bf1d719d454/volumes"
Jul 25 01:42:52.395692 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:52.395653    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"]
Jul 25 01:42:52.402911 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:52.402879    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-842d4926-tqssj"]
Jul 25 01:42:52.537474 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:52.537436    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"]
Jul 25 01:42:52.537693 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:52.537661    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" podUID="18850452-e46a-4667-8846-a95fa5da18eb" containerName="istio-proxy" containerID="cri-o://b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf" gracePeriod=30
Jul 25 01:42:53.337409 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.337319    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"]
Jul 25 01:42:53.337836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.337815    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4200e2bc-b220-4237-b298-8bf1d719d454" containerName="maas-api"
Jul 25 01:42:53.337923 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.337837    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4200e2bc-b220-4237-b298-8bf1d719d454" containerName="maas-api"
Jul 25 01:42:53.337923 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.337849    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd" containerName="revoke-keys"
Jul 25 01:42:53.337923 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.337857    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd" containerName="revoke-keys"
Jul 25 01:42:53.338111 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.337965    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4200e2bc-b220-4237-b298-8bf1d719d454" containerName="maas-api"
Jul 25 01:42:53.338111 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.337981    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd" containerName="revoke-keys"
Jul 25 01:42:53.342094 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.342070    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.344301 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.344275    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-193600d6-openshift-default-dockercfg-xwcwl\""
Jul 25 01:42:53.352679 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.352655    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"]
Jul 25 01:42:53.415024 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.414986    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bfxp\" (UniqueName: \"kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-kube-api-access-5bfxp\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.415447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.415053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-token\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.415447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.415134    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-envoy\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.415447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.415165    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-credential-socket\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.415447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.415202    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9b711c-8e28-471a-950c-e8fb73d775a4-istiod-ca-cert\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.415447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.415261    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-podinfo\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.415447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.415288    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-data\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.415447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.415316    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-certs\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.415828 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.415460    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-socket\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.516383 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-certs\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.516593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516414    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-socket\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.516593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516445    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bfxp\" (UniqueName: \"kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-kube-api-access-5bfxp\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.516593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516497    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-token\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.516593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516553    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-envoy\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.516593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516577    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-credential-socket\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.516887 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516606    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9b711c-8e28-471a-950c-e8fb73d775a4-istiod-ca-cert\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.516887 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516753    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-certs\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.516887 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516807    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-podinfo\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.516887 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516825    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-socket\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.516887 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516854    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-data\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.517148 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.516986    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-credential-socket\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.517148 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.517102    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-data\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.517283 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.517263    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9b711c-8e28-471a-950c-e8fb73d775a4-istiod-ca-cert\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.518786 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.518763    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-envoy\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.519058 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.519041    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-podinfo\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.524110 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.524082    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-token\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.524479 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.524457    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bfxp\" (UniqueName: \"kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-kube-api-access-5bfxp\") pod \"e2e-ait-193600d6-openshift-default-bf88d878-lhtlb\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") " pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.653384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.653319    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:53.815742 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.815713    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"]
Jul 25 01:42:53.818397 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:42:53.818365    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9b711c_8e28_471a_950c_e8fb73d775a4.slice/crio-ceb3a5e8e1de0bf2d2bc8eb490e10e9d4e4b22581a238c71cfa2c6b6a1867230 WatchSource:0}: Error finding container ceb3a5e8e1de0bf2d2bc8eb490e10e9d4e4b22581a238c71cfa2c6b6a1867230: Status 404 returned error can't find the container with id ceb3a5e8e1de0bf2d2bc8eb490e10e9d4e4b22581a238c71cfa2c6b6a1867230
Jul 25 01:42:53.820558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.820521    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:42:53.820659 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.820587    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:42:53.820659 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:53.820615    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:42:54.066860 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:54.066762    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" event={"ID":"5d9b711c-8e28-471a-950c-e8fb73d775a4","Type":"ContainerStarted","Data":"1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78"}
Jul 25 01:42:54.066860 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:54.066837    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" event={"ID":"5d9b711c-8e28-471a-950c-e8fb73d775a4","Type":"ContainerStarted","Data":"ceb3a5e8e1de0bf2d2bc8eb490e10e9d4e4b22581a238c71cfa2c6b6a1867230"}
Jul 25 01:42:54.089905 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:54.089857    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" podStartSLOduration=1.089840885 podStartE2EDuration="1.089840885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:42:54.088252141 +0000 UTC m=+1858.198061451" watchObservedRunningTime="2026-07-25 01:42:54.089840885 +0000 UTC m=+1858.199650281"
Jul 25 01:42:54.352994 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:54.352913    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd" path="/var/lib/kubelet/pods/aed2d0a6-79eb-45b3-bf0a-fb6ca7b4b9fd/volumes"
Jul 25 01:42:54.653950 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:54.653921    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:54.659205 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:54.659179    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:55.072128 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.072042    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:55.073002 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.072980    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:42:55.699068 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.699033    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"]
Jul 25 01:42:55.702689 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.702671    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:55.705090 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.705061    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-193600d6-serving-cert\""
Jul 25 01:42:55.705090 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.705076    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhv77\""
Jul 25 01:42:55.714718 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.714664    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"]
Jul 25 01:42:55.735710 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.735681    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwtjh\" (UniqueName: \"kubernetes.io/projected/b3dce47f-395b-46c1-a77b-af626394a0f1-kube-api-access-bwtjh\") pod \"maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:55.735859 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.735737    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3dce47f-395b-46c1-a77b-af626394a0f1-maas-api-tls\") pod \"maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:55.735925 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.735863    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3dce47f-395b-46c1-a77b-af626394a0f1-openshift-service-ca\") pod \"maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:55.836561 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.836522    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3dce47f-395b-46c1-a77b-af626394a0f1-openshift-service-ca\") pod \"maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:55.836561 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.836562    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwtjh\" (UniqueName: \"kubernetes.io/projected/b3dce47f-395b-46c1-a77b-af626394a0f1-kube-api-access-bwtjh\") pod \"maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:55.836773 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.836602    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3dce47f-395b-46c1-a77b-af626394a0f1-maas-api-tls\") pod \"maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:55.836773 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:42:55.836724    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-serving-cert: secret "maas-api-e2e-ait-193600d6-serving-cert" not found
Jul 25 01:42:55.836843 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:42:55.836785    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3dce47f-395b-46c1-a77b-af626394a0f1-maas-api-tls podName:b3dce47f-395b-46c1-a77b-af626394a0f1 nodeName:}" failed. No retries permitted until 2026-07-25 01:42:56.336766259 +0000 UTC m=+1860.446575537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b3dce47f-395b-46c1-a77b-af626394a0f1-maas-api-tls") pod "maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4" (UID: "b3dce47f-395b-46c1-a77b-af626394a0f1") : secret "maas-api-e2e-ait-193600d6-serving-cert" not found
Jul 25 01:42:55.837210 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.837187    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3dce47f-395b-46c1-a77b-af626394a0f1-openshift-service-ca\") pod \"maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:55.852890 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:55.852863    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwtjh\" (UniqueName: \"kubernetes.io/projected/b3dce47f-395b-46c1-a77b-af626394a0f1-kube-api-access-bwtjh\") pod \"maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:56.341956 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:56.341920    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3dce47f-395b-46c1-a77b-af626394a0f1-maas-api-tls\") pod \"maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:56.344212 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:56.344190    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3dce47f-395b-46c1-a77b-af626394a0f1-maas-api-tls\") pod \"maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:56.617873 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:56.617789    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhv77\""
Jul 25 01:42:56.626696 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:56.626672    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:56.761160 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:56.761138    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"]
Jul 25 01:42:56.763369 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:42:56.763339    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dce47f_395b_46c1_a77b_af626394a0f1.slice/crio-a3408e7ca3c254fae0f05c76c5b0dcb31a2809ab7b4b1c09b89d8d1a73ef7e16 WatchSource:0}: Error finding container a3408e7ca3c254fae0f05c76c5b0dcb31a2809ab7b4b1c09b89d8d1a73ef7e16: Status 404 returned error can't find the container with id a3408e7ca3c254fae0f05c76c5b0dcb31a2809ab7b4b1c09b89d8d1a73ef7e16
Jul 25 01:42:57.085851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.085789    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4" event={"ID":"b3dce47f-395b-46c1-a77b-af626394a0f1","Type":"ContainerStarted","Data":"a3408e7ca3c254fae0f05c76c5b0dcb31a2809ab7b4b1c09b89d8d1a73ef7e16"}
Jul 25 01:42:57.797682 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.797614    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:57.855173 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855140    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18850452-e46a-4667-8846-a95fa5da18eb-istiod-ca-cert\") pod \"18850452-e46a-4667-8846-a95fa5da18eb\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") "
Jul 25 01:42:57.855173 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855180    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-istio-token\") pod \"18850452-e46a-4667-8846-a95fa5da18eb\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") "
Jul 25 01:42:57.855456 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855199    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-data\") pod \"18850452-e46a-4667-8846-a95fa5da18eb\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") "
Jul 25 01:42:57.855456 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855242    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6qx9\" (UniqueName: \"kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-kube-api-access-k6qx9\") pod \"18850452-e46a-4667-8846-a95fa5da18eb\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") "
Jul 25 01:42:57.855456 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855269    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18850452-e46a-4667-8846-a95fa5da18eb-istio-podinfo\") pod \"18850452-e46a-4667-8846-a95fa5da18eb\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") "
Jul 25 01:42:57.855456 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855288    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-credential-socket\") pod \"18850452-e46a-4667-8846-a95fa5da18eb\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") "
Jul 25 01:42:57.855456 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855357    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-envoy\") pod \"18850452-e46a-4667-8846-a95fa5da18eb\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") "
Jul 25 01:42:57.855456 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855386    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-socket\") pod \"18850452-e46a-4667-8846-a95fa5da18eb\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") "
Jul 25 01:42:57.855776 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855471    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-certs\") pod \"18850452-e46a-4667-8846-a95fa5da18eb\" (UID: \"18850452-e46a-4667-8846-a95fa5da18eb\") "
Jul 25 01:42:57.855776 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855660    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18850452-e46a-4667-8846-a95fa5da18eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18850452-e46a-4667-8846-a95fa5da18eb" (UID: "18850452-e46a-4667-8846-a95fa5da18eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:42:57.855882 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855782    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18850452-e46a-4667-8846-a95fa5da18eb-istiod-ca-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:57.855882 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.855854    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18850452-e46a-4667-8846-a95fa5da18eb" (UID: "18850452-e46a-4667-8846-a95fa5da18eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:42:57.856049 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.856013    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18850452-e46a-4667-8846-a95fa5da18eb" (UID: "18850452-e46a-4667-8846-a95fa5da18eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:42:57.856129 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.856077    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18850452-e46a-4667-8846-a95fa5da18eb" (UID: "18850452-e46a-4667-8846-a95fa5da18eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:42:57.856194 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.856132    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "18850452-e46a-4667-8846-a95fa5da18eb" (UID: "18850452-e46a-4667-8846-a95fa5da18eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:42:57.857617 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.857592    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "18850452-e46a-4667-8846-a95fa5da18eb" (UID: "18850452-e46a-4667-8846-a95fa5da18eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:42:57.857727 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.857683    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18850452-e46a-4667-8846-a95fa5da18eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18850452-e46a-4667-8846-a95fa5da18eb" (UID: "18850452-e46a-4667-8846-a95fa5da18eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 25 01:42:57.857889 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.857860    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18850452-e46a-4667-8846-a95fa5da18eb" (UID: "18850452-e46a-4667-8846-a95fa5da18eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:42:57.857944 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.857898    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-kube-api-access-k6qx9" (OuterVolumeSpecName: "kube-api-access-k6qx9") pod "18850452-e46a-4667-8846-a95fa5da18eb" (UID: "18850452-e46a-4667-8846-a95fa5da18eb"). InnerVolumeSpecName "kube-api-access-k6qx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:42:57.956690 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.956660    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:57.956690 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.956686    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-istio-token\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:57.956690 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.956694    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-data\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:57.956906 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.956703    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6qx9\" (UniqueName: \"kubernetes.io/projected/18850452-e46a-4667-8846-a95fa5da18eb-kube-api-access-k6qx9\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:57.956906 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.956714    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18850452-e46a-4667-8846-a95fa5da18eb-istio-podinfo\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:57.956906 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.956722    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-credential-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:57.956906 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.956730    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-istio-envoy\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:57.956906 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:57.956739    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18850452-e46a-4667-8846-a95fa5da18eb-workload-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:42:58.092258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.092176    2572 generic.go:356] "Generic (PLEG): container finished" podID="18850452-e46a-4667-8846-a95fa5da18eb" containerID="b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf" exitCode=0
Jul 25 01:42:58.092258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.092228    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"
Jul 25 01:42:58.092462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.092264    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" event={"ID":"18850452-e46a-4667-8846-a95fa5da18eb","Type":"ContainerDied","Data":"b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf"}
Jul 25 01:42:58.092462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.092318    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk" event={"ID":"18850452-e46a-4667-8846-a95fa5da18eb","Type":"ContainerDied","Data":"a1a8cc3dfad9b403d74e0f9ab1662de80fddb4da2bb10c133639d25864e59024"}
Jul 25 01:42:58.092462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.092344    2572 scope.go:117] "RemoveContainer" containerID="b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf"
Jul 25 01:42:58.094129 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.094103    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4" event={"ID":"b3dce47f-395b-46c1-a77b-af626394a0f1","Type":"ContainerStarted","Data":"ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a"}
Jul 25 01:42:58.094222 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.094210    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:42:58.101568 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.101552    2572 scope.go:117] "RemoveContainer" containerID="db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8"
Jul 25 01:42:58.109190 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.109174    2572 scope.go:117] "RemoveContainer" containerID="b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf"
Jul 25 01:42:58.109514 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:42:58.109494    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf\": container with ID starting with b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf not found: ID does not exist" containerID="b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf"
Jul 25 01:42:58.109591 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.109521    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf"} err="failed to get container status \"b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf\": rpc error: code = NotFound desc = could not find container \"b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf\": container with ID starting with b4c13606683cacee3fb8dca1987ee7e5de112656232ef0654fb4224827ec1baf not found: ID does not exist"
Jul 25 01:42:58.109591 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.109538    2572 scope.go:117] "RemoveContainer" containerID="db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8"
Jul 25 01:42:58.109778 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:42:58.109762    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8\": container with ID starting with db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8 not found: ID does not exist" containerID="db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8"
Jul 25 01:42:58.109821 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.109784    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8"} err="failed to get container status \"db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8\": rpc error: code = NotFound desc = could not find container \"db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8\": container with ID starting with db0f15964d3dced490bb9f5fbf058f1e258626c32733ea13ca4d8ec2d8b2fcb8 not found: ID does not exist"
Jul 25 01:42:58.112367 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.112310    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4" podStartSLOduration=2.761266758 podStartE2EDuration="3.112297435s" totalImagesPullingTime="351.030677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:42:58.110725002 +0000 UTC m=+1862.220534286" watchObservedRunningTime="2026-07-25 01:42:58.112297435 +0000 UTC m=+1862.222106719"
Jul 25 01:42:58.124563 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.124542    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"]
Jul 25 01:42:58.130566 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.130541    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-842d4926-gw-openshift-default-548d9575b8-x5hxk"]
Jul 25 01:42:58.353512 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:42:58.353432    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18850452-e46a-4667-8846-a95fa5da18eb" path="/var/lib/kubelet/pods/18850452-e46a-4667-8846-a95fa5da18eb/volumes"
Jul 25 01:43:04.106441 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:04.106409    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:43:10.073215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:10.073176    2572 patch_prober.go:28] interesting pod/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 25 01:43:10.073644 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:10.073243    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 25 01:43:10.161688 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:10.161655    2572 generic.go:356] "Generic (PLEG): container finished" podID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerID="1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78" exitCode=0
Jul 25 01:43:10.161843 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:10.161729    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" event={"ID":"5d9b711c-8e28-471a-950c-e8fb73d775a4","Type":"ContainerDied","Data":"1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78"}
Jul 25 01:43:10.162124 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:10.162109    2572 scope.go:117] "RemoveContainer" containerID="1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78"
Jul 25 01:43:10.163088 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:10.163061    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:43:10.163166 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:10.163147    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:43:10.163223 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:10.163208    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:43:11.169088 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:11.169052    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" event={"ID":"5d9b711c-8e28-471a-950c-e8fb73d775a4","Type":"ContainerStarted","Data":"71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e"}
Jul 25 01:43:11.653831 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:11.653793    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:43:11.654426 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:11.654083    2572 patch_prober.go:28] interesting pod/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 25 01:43:11.654426 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:11.654151    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 25 01:43:12.654271 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:12.654227    2572 patch_prober.go:28] interesting pod/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 25 01:43:12.654884 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:12.654850    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 25 01:43:13.654614 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:13.654570    2572 patch_prober.go:28] interesting pod/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 25 01:43:13.655221 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:13.655181    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 25 01:43:14.654128 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:14.654081    2572 patch_prober.go:28] interesting pod/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Jul 25 01:43:14.654370 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:14.654152    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Jul 25 01:43:15.657881 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:15.657841    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:43:15.658277 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:15.658143    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:43:15.658813 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:15.658793    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:43:21.370824 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.370740    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"]
Jul 25 01:43:21.371258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.371061    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18850452-e46a-4667-8846-a95fa5da18eb" containerName="istio-proxy"
Jul 25 01:43:21.371258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.371071    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="18850452-e46a-4667-8846-a95fa5da18eb" containerName="istio-proxy"
Jul 25 01:43:21.371258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.371083    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18850452-e46a-4667-8846-a95fa5da18eb" containerName="istio-proxy"
Jul 25 01:43:21.371258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.371089    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="18850452-e46a-4667-8846-a95fa5da18eb" containerName="istio-proxy"
Jul 25 01:43:21.371258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.371140    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="18850452-e46a-4667-8846-a95fa5da18eb" containerName="istio-proxy"
Jul 25 01:43:21.371258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.371148    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="18850452-e46a-4667-8846-a95fa5da18eb" containerName="istio-proxy"
Jul 25 01:43:21.373245 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.373229    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"
Jul 25 01:43:21.375481 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.375465    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 01:43:21.381364 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.381305    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"]
Jul 25 01:43:21.455719 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.455688    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4832f5f4-a5ba-4787-bd07-023ac3088e74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-193600d6-zsd5k\" (UID: \"4832f5f4-a5ba-4787-bd07-023ac3088e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"
Jul 25 01:43:21.556476 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.556442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4832f5f4-a5ba-4787-bd07-023ac3088e74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-193600d6-zsd5k\" (UID: \"4832f5f4-a5ba-4787-bd07-023ac3088e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"
Jul 25 01:43:21.557052 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.557034    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4832f5f4-a5ba-4787-bd07-023ac3088e74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-193600d6-zsd5k\" (UID: \"4832f5f4-a5ba-4787-bd07-023ac3088e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"
Jul 25 01:43:21.684995 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.684966    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"
Jul 25 01:43:21.811678 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:21.811649    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"]
Jul 25 01:43:21.812830 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:43:21.812809    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4832f5f4_a5ba_4787_bd07_023ac3088e74.slice/crio-a1d8fb7d5f22814ee53dad72d5c0bc8eb842a9149c2ef81f48bd912a2afe0d8c WatchSource:0}: Error finding container a1d8fb7d5f22814ee53dad72d5c0bc8eb842a9149c2ef81f48bd912a2afe0d8c: Status 404 returned error can't find the container with id a1d8fb7d5f22814ee53dad72d5c0bc8eb842a9149c2ef81f48bd912a2afe0d8c
Jul 25 01:43:22.226787 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:22.226753    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k" event={"ID":"4832f5f4-a5ba-4787-bd07-023ac3088e74","Type":"ContainerStarted","Data":"72b1bbde3537bf0e01b77d2993a86dcef9d3aa7ee596352099fbc3be6698001f"}
Jul 25 01:43:22.227133 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:22.226816    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k" event={"ID":"4832f5f4-a5ba-4787-bd07-023ac3088e74","Type":"ContainerStarted","Data":"a1d8fb7d5f22814ee53dad72d5c0bc8eb842a9149c2ef81f48bd912a2afe0d8c"}
Jul 25 01:43:22.241184 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:22.241129    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k" podStartSLOduration=1.241116085 podStartE2EDuration="1.241116085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:43:22.240470362 +0000 UTC m=+1886.350279645" watchObservedRunningTime="2026-07-25 01:43:22.241116085 +0000 UTC m=+1886.350925369"
Jul 25 01:43:23.233653 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:23.233617    2572 generic.go:356] "Generic (PLEG): container finished" podID="4832f5f4-a5ba-4787-bd07-023ac3088e74" containerID="72b1bbde3537bf0e01b77d2993a86dcef9d3aa7ee596352099fbc3be6698001f" exitCode=0
Jul 25 01:43:23.234044 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:23.233697    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k" event={"ID":"4832f5f4-a5ba-4787-bd07-023ac3088e74","Type":"ContainerDied","Data":"72b1bbde3537bf0e01b77d2993a86dcef9d3aa7ee596352099fbc3be6698001f"}
Jul 25 01:43:24.365082 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:24.365055    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"
Jul 25 01:43:24.484168 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:24.484138    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4832f5f4-a5ba-4787-bd07-023ac3088e74-maas-api-service-ca\") pod \"4832f5f4-a5ba-4787-bd07-023ac3088e74\" (UID: \"4832f5f4-a5ba-4787-bd07-023ac3088e74\") "
Jul 25 01:43:24.484543 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:24.484520    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4832f5f4-a5ba-4787-bd07-023ac3088e74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4832f5f4-a5ba-4787-bd07-023ac3088e74" (UID: "4832f5f4-a5ba-4787-bd07-023ac3088e74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:43:24.585183 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:24.585107    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4832f5f4-a5ba-4787-bd07-023ac3088e74-maas-api-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:25.245690 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:25.245649    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k" event={"ID":"4832f5f4-a5ba-4787-bd07-023ac3088e74","Type":"ContainerDied","Data":"a1d8fb7d5f22814ee53dad72d5c0bc8eb842a9149c2ef81f48bd912a2afe0d8c"}
Jul 25 01:43:25.245858 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:25.245726    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1d8fb7d5f22814ee53dad72d5c0bc8eb842a9149c2ef81f48bd912a2afe0d8c"
Jul 25 01:43:25.245858 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:25.245666    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"
Jul 25 01:43:31.863915 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:31.863878    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"]
Jul 25 01:43:31.864387 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:31.864138    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4" podUID="b3dce47f-395b-46c1-a77b-af626394a0f1" containerName="maas-api" containerID="cri-o://ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a" gracePeriod=30
Jul 25 01:43:32.112443 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.112417    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:43:32.250253 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.250223    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwtjh\" (UniqueName: \"kubernetes.io/projected/b3dce47f-395b-46c1-a77b-af626394a0f1-kube-api-access-bwtjh\") pod \"b3dce47f-395b-46c1-a77b-af626394a0f1\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") "
Jul 25 01:43:32.250418 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.250284    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3dce47f-395b-46c1-a77b-af626394a0f1-openshift-service-ca\") pod \"b3dce47f-395b-46c1-a77b-af626394a0f1\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") "
Jul 25 01:43:32.250418 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.250355    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3dce47f-395b-46c1-a77b-af626394a0f1-maas-api-tls\") pod \"b3dce47f-395b-46c1-a77b-af626394a0f1\" (UID: \"b3dce47f-395b-46c1-a77b-af626394a0f1\") "
Jul 25 01:43:32.250659 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.250637    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3dce47f-395b-46c1-a77b-af626394a0f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b3dce47f-395b-46c1-a77b-af626394a0f1" (UID: "b3dce47f-395b-46c1-a77b-af626394a0f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:43:32.252421 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.252396    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3dce47f-395b-46c1-a77b-af626394a0f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b3dce47f-395b-46c1-a77b-af626394a0f1" (UID: "b3dce47f-395b-46c1-a77b-af626394a0f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:43:32.252528 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.252456    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3dce47f-395b-46c1-a77b-af626394a0f1-kube-api-access-bwtjh" (OuterVolumeSpecName: "kube-api-access-bwtjh") pod "b3dce47f-395b-46c1-a77b-af626394a0f1" (UID: "b3dce47f-395b-46c1-a77b-af626394a0f1"). InnerVolumeSpecName "kube-api-access-bwtjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:43:32.283225 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.283195    2572 generic.go:356] "Generic (PLEG): container finished" podID="b3dce47f-395b-46c1-a77b-af626394a0f1" containerID="ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a" exitCode=0
Jul 25 01:43:32.283387 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.283236    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"
Jul 25 01:43:32.283387 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.283274    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4" event={"ID":"b3dce47f-395b-46c1-a77b-af626394a0f1","Type":"ContainerDied","Data":"ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a"}
Jul 25 01:43:32.283387 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.283342    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4" event={"ID":"b3dce47f-395b-46c1-a77b-af626394a0f1","Type":"ContainerDied","Data":"a3408e7ca3c254fae0f05c76c5b0dcb31a2809ab7b4b1c09b89d8d1a73ef7e16"}
Jul 25 01:43:32.283387 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.283369    2572 scope.go:117] "RemoveContainer" containerID="ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a"
Jul 25 01:43:32.292466 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.292453    2572 scope.go:117] "RemoveContainer" containerID="ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a"
Jul 25 01:43:32.292731 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:43:32.292712    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a\": container with ID starting with ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a not found: ID does not exist" containerID="ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a"
Jul 25 01:43:32.292776 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.292737    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a"} err="failed to get container status \"ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a\": rpc error: code = NotFound desc = could not find container \"ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a\": container with ID starting with ef7fe2eba20d3232cbdde007cfd2027ce52321134964f8e544f31fc5f147017a not found: ID does not exist"
Jul 25 01:43:32.304120 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.304096    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"]
Jul 25 01:43:32.307132 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.307109    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-193600d6-7cbdb46f7f-w6rj4"]
Jul 25 01:43:32.350857 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.350831    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3dce47f-395b-46c1-a77b-af626394a0f1-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:32.350994 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.350861    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3dce47f-395b-46c1-a77b-af626394a0f1-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:32.350994 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.350874    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwtjh\" (UniqueName: \"kubernetes.io/projected/b3dce47f-395b-46c1-a77b-af626394a0f1-kube-api-access-bwtjh\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:32.352460 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:32.352434    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3dce47f-395b-46c1-a77b-af626394a0f1" path="/var/lib/kubelet/pods/b3dce47f-395b-46c1-a77b-af626394a0f1/volumes"
Jul 25 01:43:36.443933 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:36.443900    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"]
Jul 25 01:43:36.445961 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:36.445933    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-193600d6-zsd5k"]
Jul 25 01:43:37.693218 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:37.693185    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"]
Jul 25 01:43:37.693721 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:37.693434    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy" containerID="cri-o://71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e" gracePeriod=30
Jul 25 01:43:38.352928 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.352897    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4832f5f4-a5ba-4787-bd07-023ac3088e74" path="/var/lib/kubelet/pods/4832f5f4-a5ba-4787-bd07-023ac3088e74/volumes"
Jul 25 01:43:38.487500 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.487459    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"]
Jul 25 01:43:38.487928 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.487914    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3dce47f-395b-46c1-a77b-af626394a0f1" containerName="maas-api"
Jul 25 01:43:38.487975 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.487931    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3dce47f-395b-46c1-a77b-af626394a0f1" containerName="maas-api"
Jul 25 01:43:38.488014 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.487979    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4832f5f4-a5ba-4787-bd07-023ac3088e74" containerName="revoke-keys"
Jul 25 01:43:38.488014 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.487989    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4832f5f4-a5ba-4787-bd07-023ac3088e74" containerName="revoke-keys"
Jul 25 01:43:38.488094 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.488083    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3dce47f-395b-46c1-a77b-af626394a0f1" containerName="maas-api"
Jul 25 01:43:38.488128 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.488100    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4832f5f4-a5ba-4787-bd07-023ac3088e74" containerName="revoke-keys"
Jul 25 01:43:38.498306 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.498279    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.501060 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.501031    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-906a2829-openshift-default-dockercfg-5n2cn\""
Jul 25 01:43:38.503836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.503809    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"]
Jul 25 01:43:38.606696 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.606597    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istiod-ca-cert\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.606696 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.606642    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrd29\" (UniqueName: \"kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-kube-api-access-vrd29\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.606696 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.606663    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-credential-socket\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.606968 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.606712    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-envoy\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.606968 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.606731    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-certs\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.606968 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.606754    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-token\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.606968 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.606864    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-socket\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.606968 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.606907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-data\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.607188 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.606978    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-podinfo\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.707628 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.707600    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-envoy\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708101 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.707634    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-certs\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708101 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.707656    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-token\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708101 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.707788    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-socket\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708101 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.707839    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-data\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708101 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.707889    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-podinfo\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708412 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.708139    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istiod-ca-cert\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708412 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.708194    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-socket\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708412 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.708196    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrd29\" (UniqueName: \"kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-kube-api-access-vrd29\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708412 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.708253    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-credential-socket\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708641 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.708591    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-credential-socket\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708704 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.708640    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-data\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708752 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.708699    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-certs\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.708816 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.708796    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istiod-ca-cert\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.710158 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.710134    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-envoy\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.710674 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.710654    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-podinfo\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.715418 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.715395    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-token\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.715604 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.715583    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrd29\" (UniqueName: \"kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-kube-api-access-vrd29\") pod \"e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") " pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.811167 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.811131    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:38.948962 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.948933    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"]
Jul 25 01:43:38.950924 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:43:38.950897    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d5ed19_9d8d_4838_b3b9_284b5850e65f.slice/crio-29c531cb1eb0e0b89169c8d84a32e2ce5958906f2db5527367449294ce84a92e WatchSource:0}: Error finding container 29c531cb1eb0e0b89169c8d84a32e2ce5958906f2db5527367449294ce84a92e: Status 404 returned error can't find the container with id 29c531cb1eb0e0b89169c8d84a32e2ce5958906f2db5527367449294ce84a92e
Jul 25 01:43:38.953049 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.953020    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:43:38.953136 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.953082    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:43:38.953136 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:38.953114    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:43:39.320713 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:39.320673    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" event={"ID":"64d5ed19-9d8d-4838-b3b9-284b5850e65f","Type":"ContainerStarted","Data":"6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07"}
Jul 25 01:43:39.320901 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:39.320733    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" event={"ID":"64d5ed19-9d8d-4838-b3b9-284b5850e65f","Type":"ContainerStarted","Data":"29c531cb1eb0e0b89169c8d84a32e2ce5958906f2db5527367449294ce84a92e"}
Jul 25 01:43:39.341769 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:39.341723    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" podStartSLOduration=1.341707909 podStartE2EDuration="1.341707909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:43:39.339679861 +0000 UTC m=+1903.449489146" watchObservedRunningTime="2026-07-25 01:43:39.341707909 +0000 UTC m=+1903.451517193"
Jul 25 01:43:39.811817 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:39.811780    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:39.817373 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:39.817319    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:40.325648 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.325610    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:40.326390 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.326371    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:40.762717 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.762682    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"]
Jul 25 01:43:40.765080 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.765058    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:40.767350 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.767299    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-906a2829-serving-cert\""
Jul 25 01:43:40.767475 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.767380    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhv77\""
Jul 25 01:43:40.775904 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.775876    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"]
Jul 25 01:43:40.824230 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.824193    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsnqr\" (UniqueName: \"kubernetes.io/projected/9f01fced-916f-4c21-986a-4ab1a9bb39f3-kube-api-access-tsnqr\") pod \"maas-api-e2e-derive-906a2829-667855bf96-25qjf\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:40.824604 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.824257    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls\") pod \"maas-api-e2e-derive-906a2829-667855bf96-25qjf\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:40.824604 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.824317    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f01fced-916f-4c21-986a-4ab1a9bb39f3-openshift-service-ca\") pod \"maas-api-e2e-derive-906a2829-667855bf96-25qjf\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:40.925071 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.925034    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f01fced-916f-4c21-986a-4ab1a9bb39f3-openshift-service-ca\") pod \"maas-api-e2e-derive-906a2829-667855bf96-25qjf\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:40.925281 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.925097    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsnqr\" (UniqueName: \"kubernetes.io/projected/9f01fced-916f-4c21-986a-4ab1a9bb39f3-kube-api-access-tsnqr\") pod \"maas-api-e2e-derive-906a2829-667855bf96-25qjf\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:40.925281 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.925126    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls\") pod \"maas-api-e2e-derive-906a2829-667855bf96-25qjf\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:40.925281 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:43:40.925229    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-serving-cert: secret "maas-api-e2e-derive-906a2829-serving-cert" not found
Jul 25 01:43:40.925281 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:43:40.925283    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls podName:9f01fced-916f-4c21-986a-4ab1a9bb39f3 nodeName:}" failed. No retries permitted until 2026-07-25 01:43:41.425267911 +0000 UTC m=+1905.535077173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls") pod "maas-api-e2e-derive-906a2829-667855bf96-25qjf" (UID: "9f01fced-916f-4c21-986a-4ab1a9bb39f3") : secret "maas-api-e2e-derive-906a2829-serving-cert" not found
Jul 25 01:43:40.925799 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.925775    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f01fced-916f-4c21-986a-4ab1a9bb39f3-openshift-service-ca\") pod \"maas-api-e2e-derive-906a2829-667855bf96-25qjf\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:40.932945 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:40.932923    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsnqr\" (UniqueName: \"kubernetes.io/projected/9f01fced-916f-4c21-986a-4ab1a9bb39f3-kube-api-access-tsnqr\") pod \"maas-api-e2e-derive-906a2829-667855bf96-25qjf\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:41.429235 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:41.429196    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls\") pod \"maas-api-e2e-derive-906a2829-667855bf96-25qjf\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:41.429461 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:43:41.429413    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-serving-cert: secret "maas-api-e2e-derive-906a2829-serving-cert" not found
Jul 25 01:43:41.429541 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:43:41.429484    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls podName:9f01fced-916f-4c21-986a-4ab1a9bb39f3 nodeName:}" failed. No retries permitted until 2026-07-25 01:43:42.429464677 +0000 UTC m=+1906.539273939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls") pod "maas-api-e2e-derive-906a2829-667855bf96-25qjf" (UID: "9f01fced-916f-4c21-986a-4ab1a9bb39f3") : secret "maas-api-e2e-derive-906a2829-serving-cert" not found
Jul 25 01:43:42.438226 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.438188    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls\") pod \"maas-api-e2e-derive-906a2829-667855bf96-25qjf\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:42.440607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.440584    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls\") pod \"maas-api-e2e-derive-906a2829-667855bf96-25qjf\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:42.576777 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.576738    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:42.914279 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.914252    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"]
Jul 25 01:43:42.916901 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:43:42.916871    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f01fced_916f_4c21_986a_4ab1a9bb39f3.slice/crio-e36528287c2993b0fb76988bc9d23e6e7cc95b075b02a2f6288936f05ffc8d22 WatchSource:0}: Error finding container e36528287c2993b0fb76988bc9d23e6e7cc95b075b02a2f6288936f05ffc8d22: Status 404 returned error can't find the container with id e36528287c2993b0fb76988bc9d23e6e7cc95b075b02a2f6288936f05ffc8d22
Jul 25 01:43:42.934337 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.934307    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:43:42.941855 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.941826    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-podinfo\") pod \"5d9b711c-8e28-471a-950c-e8fb73d775a4\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") "
Jul 25 01:43:42.941972 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.941866    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-envoy\") pod \"5d9b711c-8e28-471a-950c-e8fb73d775a4\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") "
Jul 25 01:43:42.941972 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.941891    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9b711c-8e28-471a-950c-e8fb73d775a4-istiod-ca-cert\") pod \"5d9b711c-8e28-471a-950c-e8fb73d775a4\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") "
Jul 25 01:43:42.941972 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.941942    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-credential-socket\") pod \"5d9b711c-8e28-471a-950c-e8fb73d775a4\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") "
Jul 25 01:43:42.942144 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.941974    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bfxp\" (UniqueName: \"kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-kube-api-access-5bfxp\") pod \"5d9b711c-8e28-471a-950c-e8fb73d775a4\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") "
Jul 25 01:43:42.942144 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.942029    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-certs\") pod \"5d9b711c-8e28-471a-950c-e8fb73d775a4\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") "
Jul 25 01:43:42.942144 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.942058    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-token\") pod \"5d9b711c-8e28-471a-950c-e8fb73d775a4\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") "
Jul 25 01:43:42.942144 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.942111    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-socket\") pod \"5d9b711c-8e28-471a-950c-e8fb73d775a4\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") "
Jul 25 01:43:42.942144 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.942141    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-data\") pod \"5d9b711c-8e28-471a-950c-e8fb73d775a4\" (UID: \"5d9b711c-8e28-471a-950c-e8fb73d775a4\") "
Jul 25 01:43:42.942412 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.942204    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d9b711c-8e28-471a-950c-e8fb73d775a4" (UID: "5d9b711c-8e28-471a-950c-e8fb73d775a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:43:42.942494 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.942299    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9b711c-8e28-471a-950c-e8fb73d775a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d9b711c-8e28-471a-950c-e8fb73d775a4" (UID: "5d9b711c-8e28-471a-950c-e8fb73d775a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:43:42.942809 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.942640    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-credential-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:42.942893 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.942817    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9b711c-8e28-471a-950c-e8fb73d775a4-istiod-ca-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:42.942893 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.942816    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d9b711c-8e28-471a-950c-e8fb73d775a4" (UID: "5d9b711c-8e28-471a-950c-e8fb73d775a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:43:42.942893 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.942833    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d9b711c-8e28-471a-950c-e8fb73d775a4" (UID: "5d9b711c-8e28-471a-950c-e8fb73d775a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:43:42.943081 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.943058    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d9b711c-8e28-471a-950c-e8fb73d775a4" (UID: "5d9b711c-8e28-471a-950c-e8fb73d775a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:43:42.944149 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.944125    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d9b711c-8e28-471a-950c-e8fb73d775a4" (UID: "5d9b711c-8e28-471a-950c-e8fb73d775a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 25 01:43:42.944566 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.944549    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d9b711c-8e28-471a-950c-e8fb73d775a4" (UID: "5d9b711c-8e28-471a-950c-e8fb73d775a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:43:42.944655 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.944573    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d9b711c-8e28-471a-950c-e8fb73d775a4" (UID: "5d9b711c-8e28-471a-950c-e8fb73d775a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:43:42.944895 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:42.944879    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-kube-api-access-5bfxp" (OuterVolumeSpecName: "kube-api-access-5bfxp") pod "5d9b711c-8e28-471a-950c-e8fb73d775a4" (UID: "5d9b711c-8e28-471a-950c-e8fb73d775a4"). InnerVolumeSpecName "kube-api-access-5bfxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:43:43.043307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.043234    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:43.043307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.043260    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-token\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:43.043307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.043269    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-workload-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:43.043307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.043280    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-data\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:43.043307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.043293    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-podinfo\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:43.043307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.043301    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9b711c-8e28-471a-950c-e8fb73d775a4-istio-envoy\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:43.043307 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.043309    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bfxp\" (UniqueName: \"kubernetes.io/projected/5d9b711c-8e28-471a-950c-e8fb73d775a4-kube-api-access-5bfxp\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:43:43.342720 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.342682    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf" event={"ID":"9f01fced-916f-4c21-986a-4ab1a9bb39f3","Type":"ContainerStarted","Data":"7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447"}
Jul 25 01:43:43.342876 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.342786    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:43.342876 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.342812    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf" event={"ID":"9f01fced-916f-4c21-986a-4ab1a9bb39f3","Type":"ContainerStarted","Data":"e36528287c2993b0fb76988bc9d23e6e7cc95b075b02a2f6288936f05ffc8d22"}
Jul 25 01:43:43.344504 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.344478    2572 generic.go:356] "Generic (PLEG): container finished" podID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerID="71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e" exitCode=0
Jul 25 01:43:43.344619 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.344521    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" event={"ID":"5d9b711c-8e28-471a-950c-e8fb73d775a4","Type":"ContainerDied","Data":"71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e"}
Jul 25 01:43:43.344619 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.344556    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb" event={"ID":"5d9b711c-8e28-471a-950c-e8fb73d775a4","Type":"ContainerDied","Data":"ceb3a5e8e1de0bf2d2bc8eb490e10e9d4e4b22581a238c71cfa2c6b6a1867230"}
Jul 25 01:43:43.344619 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.344555    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"
Jul 25 01:43:43.344619 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.344572    2572 scope.go:117] "RemoveContainer" containerID="71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e"
Jul 25 01:43:43.357064 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.357043    2572 scope.go:117] "RemoveContainer" containerID="1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78"
Jul 25 01:43:43.359991 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.359947    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf" podStartSLOduration=3.022313546 podStartE2EDuration="3.359932844s" totalImagesPullingTime="337.619298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:43:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:43:43.358149565 +0000 UTC m=+1907.467958866" watchObservedRunningTime="2026-07-25 01:43:43.359932844 +0000 UTC m=+1907.469742181"
Jul 25 01:43:43.366115 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.366095    2572 scope.go:117] "RemoveContainer" containerID="71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e"
Jul 25 01:43:43.366402 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:43:43.366381    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e\": container with ID starting with 71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e not found: ID does not exist" containerID="71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e"
Jul 25 01:43:43.366469 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.366410    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e"} err="failed to get container status \"71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e\": rpc error: code = NotFound desc = could not find container \"71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e\": container with ID starting with 71874e664de9e5c793fc3504386db8fd94a66dd559a8f36c508d09abe449013e not found: ID does not exist"
Jul 25 01:43:43.366469 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.366427    2572 scope.go:117] "RemoveContainer" containerID="1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78"
Jul 25 01:43:43.366665 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:43:43.366645    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78\": container with ID starting with 1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78 not found: ID does not exist" containerID="1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78"
Jul 25 01:43:43.366725 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.366674    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78"} err="failed to get container status \"1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78\": rpc error: code = NotFound desc = could not find container \"1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78\": container with ID starting with 1271cdb9b424d85e5a8ea6faa825d0439efb9c93c5c06713a50517799cd3cb78 not found: ID does not exist"
Jul 25 01:43:43.371117 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.371095    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"]
Jul 25 01:43:43.376718 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:43.376700    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-193600d6-openshift-default-bf88d878-lhtlb"]
Jul 25 01:43:44.352853 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:44.352822    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" path="/var/lib/kubelet/pods/5d9b711c-8e28-471a-950c-e8fb73d775a4/volumes"
Jul 25 01:43:49.362826 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:49.362784    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:43:53.401983 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:53.401948    2572 generic.go:356] "Generic (PLEG): container finished" podID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerID="6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07" exitCode=0
Jul 25 01:43:53.402411 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:53.402025    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" event={"ID":"64d5ed19-9d8d-4838-b3b9-284b5850e65f","Type":"ContainerDied","Data":"6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07"}
Jul 25 01:43:53.402469 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:53.402455    2572 scope.go:117] "RemoveContainer" containerID="6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07"
Jul 25 01:43:53.403489 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:53.403465    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:43:53.403539 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:53.403531    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:43:53.403580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:53.403560    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:43:54.409867 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:54.409833    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" event={"ID":"64d5ed19-9d8d-4838-b3b9-284b5850e65f","Type":"ContainerStarted","Data":"866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12"}
Jul 25 01:43:54.812092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:54.811999    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:54.813183 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:54.813153    2572 patch_prober.go:28] interesting pod/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 25 01:43:54.813305 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:54.813207    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 25 01:43:55.325987 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:55.325952    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:55.811859 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:55.811816    2572 patch_prober.go:28] interesting pod/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 25 01:43:55.812381 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:55.811878    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 25 01:43:56.811736 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:56.811633    2572 patch_prober.go:28] interesting pod/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 25 01:43:56.811736 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:56.811692    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 25 01:43:57.811992 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:57.811953    2572 patch_prober.go:28] interesting pod/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Jul 25 01:43:57.812417 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:57.812026    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Jul 25 01:43:58.815391 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:58.815357    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:43:58.816407 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:43:58.816387    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:44:16.087517 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.087485    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"]
Jul 25 01:44:16.088040 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.087698    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf" podUID="9f01fced-916f-4c21-986a-4ab1a9bb39f3" containerName="maas-api" containerID="cri-o://7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447" gracePeriod=30
Jul 25 01:44:16.329297 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.329255    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:44:16.431804 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.431776    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls\") pod \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") "
Jul 25 01:44:16.431959 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.431827    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f01fced-916f-4c21-986a-4ab1a9bb39f3-openshift-service-ca\") pod \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") "
Jul 25 01:44:16.431959 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.431879    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsnqr\" (UniqueName: \"kubernetes.io/projected/9f01fced-916f-4c21-986a-4ab1a9bb39f3-kube-api-access-tsnqr\") pod \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\" (UID: \"9f01fced-916f-4c21-986a-4ab1a9bb39f3\") "
Jul 25 01:44:16.432262 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.432237    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f01fced-916f-4c21-986a-4ab1a9bb39f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f01fced-916f-4c21-986a-4ab1a9bb39f3" (UID: "9f01fced-916f-4c21-986a-4ab1a9bb39f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:44:16.433796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.433777    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f01fced-916f-4c21-986a-4ab1a9bb39f3" (UID: "9f01fced-916f-4c21-986a-4ab1a9bb39f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:44:16.433914 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.433892    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f01fced-916f-4c21-986a-4ab1a9bb39f3-kube-api-access-tsnqr" (OuterVolumeSpecName: "kube-api-access-tsnqr") pod "9f01fced-916f-4c21-986a-4ab1a9bb39f3" (UID: "9f01fced-916f-4c21-986a-4ab1a9bb39f3"). InnerVolumeSpecName "kube-api-access-tsnqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:44:16.522688 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.522657    2572 generic.go:356] "Generic (PLEG): container finished" podID="9f01fced-916f-4c21-986a-4ab1a9bb39f3" containerID="7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447" exitCode=0
Jul 25 01:44:16.522836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.522698    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"
Jul 25 01:44:16.522836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.522745    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf" event={"ID":"9f01fced-916f-4c21-986a-4ab1a9bb39f3","Type":"ContainerDied","Data":"7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447"}
Jul 25 01:44:16.522836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.522799    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf" event={"ID":"9f01fced-916f-4c21-986a-4ab1a9bb39f3","Type":"ContainerDied","Data":"e36528287c2993b0fb76988bc9d23e6e7cc95b075b02a2f6288936f05ffc8d22"}
Jul 25 01:44:16.522836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.522821    2572 scope.go:117] "RemoveContainer" containerID="7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447"
Jul 25 01:44:16.532135 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.532120    2572 scope.go:117] "RemoveContainer" containerID="7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447"
Jul 25 01:44:16.532396 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:44:16.532377    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447\": container with ID starting with 7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447 not found: ID does not exist" containerID="7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447"
Jul 25 01:44:16.532471 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.532403    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447"} err="failed to get container status \"7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447\": rpc error: code = NotFound desc = could not find container \"7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447\": container with ID starting with 7b409b1b9274efa11140b62a347adc1a57ca70a2577f23f95fee3115ff8de447 not found: ID does not exist"
Jul 25 01:44:16.532574 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.532557    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f01fced-916f-4c21-986a-4ab1a9bb39f3-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:16.532615 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.532577    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f01fced-916f-4c21-986a-4ab1a9bb39f3-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:16.532615 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.532588    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsnqr\" (UniqueName: \"kubernetes.io/projected/9f01fced-916f-4c21-986a-4ab1a9bb39f3-kube-api-access-tsnqr\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:16.550352 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.550308    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"]
Jul 25 01:44:16.550543 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:16.550528    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-906a2829-667855bf96-25qjf"]
Jul 25 01:44:18.353008 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:18.352975    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f01fced-916f-4c21-986a-4ab1a9bb39f3" path="/var/lib/kubelet/pods/9f01fced-916f-4c21-986a-4ab1a9bb39f3/volumes"
Jul 25 01:44:20.819424 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:20.819389    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"]
Jul 25 01:44:20.819834 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:20.819670    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerName="istio-proxy" containerID="cri-o://866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12" gracePeriod=30
Jul 25 01:44:26.070041 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.070012    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:44:26.214369 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214339    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-envoy\") pod \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") "
Jul 25 01:44:26.214369 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214373    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-token\") pod \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") "
Jul 25 01:44:26.214622 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214390    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-socket\") pod \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") "
Jul 25 01:44:26.214622 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214423    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-certs\") pod \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") "
Jul 25 01:44:26.214622 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214546    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-podinfo\") pod \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") "
Jul 25 01:44:26.214622 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214620    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-credential-socket\") pod \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") "
Jul 25 01:44:26.214851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214648    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64d5ed19-9d8d-4838-b3b9-284b5850e65f" (UID: "64d5ed19-9d8d-4838-b3b9-284b5850e65f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:44:26.214851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214678    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-data\") pod \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") "
Jul 25 01:44:26.214851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214725    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrd29\" (UniqueName: \"kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-kube-api-access-vrd29\") pod \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") "
Jul 25 01:44:26.214851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214749    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64d5ed19-9d8d-4838-b3b9-284b5850e65f" (UID: "64d5ed19-9d8d-4838-b3b9-284b5850e65f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:44:26.214851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214756    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istiod-ca-cert\") pod \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\" (UID: \"64d5ed19-9d8d-4838-b3b9-284b5850e65f\") "
Jul 25 01:44:26.215110 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.214894    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64d5ed19-9d8d-4838-b3b9-284b5850e65f" (UID: "64d5ed19-9d8d-4838-b3b9-284b5850e65f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:44:26.215110 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.215060    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64d5ed19-9d8d-4838-b3b9-284b5850e65f" (UID: "64d5ed19-9d8d-4838-b3b9-284b5850e65f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:44:26.215110 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.215087    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:26.215110 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.215106    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-workload-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:26.215350 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.215120    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-credential-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:26.215631 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.215605    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-data" (OuterVolumeSpecName: "istio-data") pod "64d5ed19-9d8d-4838-b3b9-284b5850e65f" (UID: "64d5ed19-9d8d-4838-b3b9-284b5850e65f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:44:26.216621 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.216596    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64d5ed19-9d8d-4838-b3b9-284b5850e65f" (UID: "64d5ed19-9d8d-4838-b3b9-284b5850e65f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:44:26.216735 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.216674    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-token" (OuterVolumeSpecName: "istio-token") pod "64d5ed19-9d8d-4838-b3b9-284b5850e65f" (UID: "64d5ed19-9d8d-4838-b3b9-284b5850e65f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:44:26.216735 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.216676    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64d5ed19-9d8d-4838-b3b9-284b5850e65f" (UID: "64d5ed19-9d8d-4838-b3b9-284b5850e65f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 25 01:44:26.216923 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.216877    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-kube-api-access-vrd29" (OuterVolumeSpecName: "kube-api-access-vrd29") pod "64d5ed19-9d8d-4838-b3b9-284b5850e65f" (UID: "64d5ed19-9d8d-4838-b3b9-284b5850e65f"). InnerVolumeSpecName "kube-api-access-vrd29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:44:26.315660 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.315634    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-podinfo\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:26.315660 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.315659    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-data\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:26.315815 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.315668    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrd29\" (UniqueName: \"kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-kube-api-access-vrd29\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:26.315815 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.315680    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istiod-ca-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:26.315815 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.315688    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-envoy\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:26.315815 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.315696    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64d5ed19-9d8d-4838-b3b9-284b5850e65f-istio-token\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:44:26.577282 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.577197    2572 generic.go:356] "Generic (PLEG): container finished" podID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerID="866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12" exitCode=0
Jul 25 01:44:26.577282 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.577262    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"
Jul 25 01:44:26.577538 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.577286    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" event={"ID":"64d5ed19-9d8d-4838-b3b9-284b5850e65f","Type":"ContainerDied","Data":"866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12"}
Jul 25 01:44:26.577538 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.577371    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz" event={"ID":"64d5ed19-9d8d-4838-b3b9-284b5850e65f","Type":"ContainerDied","Data":"29c531cb1eb0e0b89169c8d84a32e2ce5958906f2db5527367449294ce84a92e"}
Jul 25 01:44:26.577538 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.577405    2572 scope.go:117] "RemoveContainer" containerID="866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12"
Jul 25 01:44:26.586452 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.586434    2572 scope.go:117] "RemoveContainer" containerID="6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07"
Jul 25 01:44:26.594259 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.594242    2572 scope.go:117] "RemoveContainer" containerID="866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12"
Jul 25 01:44:26.594545 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:44:26.594527    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12\": container with ID starting with 866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12 not found: ID does not exist" containerID="866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12"
Jul 25 01:44:26.594610 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.594555    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12"} err="failed to get container status \"866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12\": rpc error: code = NotFound desc = could not find container \"866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12\": container with ID starting with 866eaf52a3dacda057ef20444a5390f39f487ed747731ad0e4bebfb991ab6f12 not found: ID does not exist"
Jul 25 01:44:26.594610 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.594572    2572 scope.go:117] "RemoveContainer" containerID="6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07"
Jul 25 01:44:26.594840 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:44:26.594822    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07\": container with ID starting with 6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07 not found: ID does not exist" containerID="6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07"
Jul 25 01:44:26.594891 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.594846    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07"} err="failed to get container status \"6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07\": rpc error: code = NotFound desc = could not find container \"6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07\": container with ID starting with 6cd5736d2cbf1dfd5e83badd5433fdf360bdbfa706dd550a7d0a8f68efddaa07 not found: ID does not exist"
Jul 25 01:44:26.595851 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.595832    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"]
Jul 25 01:44:26.600233 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:26.600213    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-906a2829-openshift-default-5f7bc5cd7b-wksbz"]
Jul 25 01:44:28.352852 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:44:28.352817    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" path="/var/lib/kubelet/pods/64d5ed19-9d8d-4838-b3b9-284b5850e65f/volumes"
Jul 25 01:45:00.133791 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.133741    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv"]
Jul 25 01:45:00.134888 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.134854    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy"
Jul 25 01:45:00.134888 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.134887    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy"
Jul 25 01:45:00.135092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.134906    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerName="istio-proxy"
Jul 25 01:45:00.135092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.134916    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerName="istio-proxy"
Jul 25 01:45:00.135092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.134930    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerName="istio-proxy"
Jul 25 01:45:00.135092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.134938    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerName="istio-proxy"
Jul 25 01:45:00.135092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.134954    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f01fced-916f-4c21-986a-4ab1a9bb39f3" containerName="maas-api"
Jul 25 01:45:00.135092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.134963    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f01fced-916f-4c21-986a-4ab1a9bb39f3" containerName="maas-api"
Jul 25 01:45:00.135092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.135044    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerName="istio-proxy"
Jul 25 01:45:00.135092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.135054    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy"
Jul 25 01:45:00.135092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.135064    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="64d5ed19-9d8d-4838-b3b9-284b5850e65f" containerName="istio-proxy"
Jul 25 01:45:00.135092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.135076    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f01fced-916f-4c21-986a-4ab1a9bb39f3" containerName="maas-api"
Jul 25 01:45:00.135092 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.135083    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy"
Jul 25 01:45:00.138440 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.138416    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv"
Jul 25 01:45:00.141436 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.141229    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 01:45:00.147362 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.147288    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv"]
Jul 25 01:45:00.155545 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.155519    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv"
Jul 25 01:45:00.298790 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.298724    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv"]
Jul 25 01:45:00.301348 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:45:00.301300    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99541586_a0e1_4f0f_8ce4_f8ee7c8cd843.slice/crio-173fe6e7e4c78c816bf61a8aef606ab8301bf2a6dfa6b294cd3472ce99808ee1 WatchSource:0}: Error finding container 173fe6e7e4c78c816bf61a8aef606ab8301bf2a6dfa6b294cd3472ce99808ee1: Status 404 returned error can't find the container with id 173fe6e7e4c78c816bf61a8aef606ab8301bf2a6dfa6b294cd3472ce99808ee1
Jul 25 01:45:00.754554 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.754447    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv" event={"ID":"99541586-a0e1-4f0f-8ce4-f8ee7c8cd843","Type":"ContainerStarted","Data":"f393e6ba1f7ab3c6113cd8bc9de7b4da1373e321da06ba4a196d61281fbde819"}
Jul 25 01:45:00.754554 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.754552    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv" event={"ID":"99541586-a0e1-4f0f-8ce4-f8ee7c8cd843","Type":"ContainerStarted","Data":"173fe6e7e4c78c816bf61a8aef606ab8301bf2a6dfa6b294cd3472ce99808ee1"}
Jul 25 01:45:00.770354 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:00.770283    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv" podStartSLOduration=0.770265113 podStartE2EDuration="770.265113ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:45:00.770168026 +0000 UTC m=+1984.879977336" watchObservedRunningTime="2026-07-25 01:45:00.770265113 +0000 UTC m=+1984.880074397"
Jul 25 01:45:01.761034 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:01.760995    2572 generic.go:356] "Generic (PLEG): container finished" podID="99541586-a0e1-4f0f-8ce4-f8ee7c8cd843" containerID="f393e6ba1f7ab3c6113cd8bc9de7b4da1373e321da06ba4a196d61281fbde819" exitCode=0
Jul 25 01:45:01.761443 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:01.761083    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv" event={"ID":"99541586-a0e1-4f0f-8ce4-f8ee7c8cd843","Type":"ContainerDied","Data":"f393e6ba1f7ab3c6113cd8bc9de7b4da1373e321da06ba4a196d61281fbde819"}
Jul 25 01:45:02.893942 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:02.893916    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv"
Jul 25 01:45:03.771911 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:03.771882    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv"
Jul 25 01:45:03.771911 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:03.771890    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749065-6wxnv" event={"ID":"99541586-a0e1-4f0f-8ce4-f8ee7c8cd843","Type":"ContainerDied","Data":"173fe6e7e4c78c816bf61a8aef606ab8301bf2a6dfa6b294cd3472ce99808ee1"}
Jul 25 01:45:03.772133 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:45:03.771934    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="173fe6e7e4c78c816bf61a8aef606ab8301bf2a6dfa6b294cd3472ce99808ee1"
Jul 25 01:46:09.792193 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.792141    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"]
Jul 25 01:46:09.792965 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.792939    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy"
Jul 25 01:46:09.793018 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.792969    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d9b711c-8e28-471a-950c-e8fb73d775a4" containerName="istio-proxy"
Jul 25 01:46:09.793018 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.792997    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99541586-a0e1-4f0f-8ce4-f8ee7c8cd843" containerName="cleanup"
Jul 25 01:46:09.793018 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.793005    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="99541586-a0e1-4f0f-8ce4-f8ee7c8cd843" containerName="cleanup"
Jul 25 01:46:09.793112 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.793084    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="99541586-a0e1-4f0f-8ce4-f8ee7c8cd843" containerName="cleanup"
Jul 25 01:46:09.796144 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.796123    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.799586 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.799556    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 25 01:46:09.799698 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.799556    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6d92f36d-openshift-default-dockercfg-rt4d6\""
Jul 25 01:46:09.800718 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.800697    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"]
Jul 25 01:46:09.859449 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.859419    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-socket\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.859595 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.859462    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-istio-token\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.859595 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.859508    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-envoy\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.859595 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.859531    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54519780-4c42-460d-a920-6229ce71cfeb-istio-podinfo\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.859595 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.859546    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6n5\" (UniqueName: \"kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-kube-api-access-hs6n5\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.859747 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.859625    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-credential-socket\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.859747 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.859654    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-certs\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.859747 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.859675    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-data\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.859747 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.859690    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54519780-4c42-460d-a920-6229ce71cfeb-istiod-ca-cert\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.960461 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.960427    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-istio-token\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.960641 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.960483    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-envoy\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.960641 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.960500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54519780-4c42-460d-a920-6229ce71cfeb-istio-podinfo\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.960641 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.960518    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6n5\" (UniqueName: \"kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-kube-api-access-hs6n5\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.960641 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.960539    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-credential-socket\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.960641 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.960562    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-certs\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.960960 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.960686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-data\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.960960 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.960725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54519780-4c42-460d-a920-6229ce71cfeb-istiod-ca-cert\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.960960 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.960812    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-socket\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.961118 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.960963    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-credential-socket\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.961118 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.961025    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-data\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.961220 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.961119    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-socket\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.961336 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.961281    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-certs\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.961485 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.961468    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54519780-4c42-460d-a920-6229ce71cfeb-istiod-ca-cert\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.962748 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.962730    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-envoy\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.962962 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.962943    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54519780-4c42-460d-a920-6229ce71cfeb-istio-podinfo\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.968350 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.968291    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-istio-token\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:09.968491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:09.968468    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6n5\" (UniqueName: \"kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-kube-api-access-hs6n5\") pod \"e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") " pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:10.107098 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:10.107026    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:10.242489 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:10.242455    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"]
Jul 25 01:46:10.243500 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:46:10.243474    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54519780_4c42_460d_a920_6229ce71cfeb.slice/crio-80b4fce168ea1c0360573d44bb06a9926c70ab37446ad1af92f69738e52b5833 WatchSource:0}: Error finding container 80b4fce168ea1c0360573d44bb06a9926c70ab37446ad1af92f69738e52b5833: Status 404 returned error can't find the container with id 80b4fce168ea1c0360573d44bb06a9926c70ab37446ad1af92f69738e52b5833
Jul 25 01:46:10.245646 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:10.245610    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:46:10.245735 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:10.245693    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:46:10.245775 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:10.245733    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:46:11.117688 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:11.117649    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7" event={"ID":"54519780-4c42-460d-a920-6229ce71cfeb","Type":"ContainerStarted","Data":"3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077"}
Jul 25 01:46:11.118082 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:11.117706    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7" event={"ID":"54519780-4c42-460d-a920-6229ce71cfeb","Type":"ContainerStarted","Data":"80b4fce168ea1c0360573d44bb06a9926c70ab37446ad1af92f69738e52b5833"}
Jul 25 01:46:11.136355 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:11.136290    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7" podStartSLOduration=2.136275928 podStartE2EDuration="2.136275928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:46:11.134804733 +0000 UTC m=+2055.244614052" watchObservedRunningTime="2026-07-25 01:46:11.136275928 +0000 UTC m=+2055.246085212"
Jul 25 01:46:12.107206 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:12.107162    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:12.111970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:12.111945    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:12.122350 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:12.122310    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:12.123153 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:12.123136    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:13.352126 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.352095    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"]
Jul 25 01:46:13.355575 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.355553    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:13.358418 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.358395    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6d92f36d-serving-cert\""
Jul 25 01:46:13.358570 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.358553    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhv77\""
Jul 25 01:46:13.380939 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.380906    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"]
Jul 25 01:46:13.494300 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.494255    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efb5861a-ce54-4654-b978-95e55325bdb5-openshift-service-ca\") pod \"maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:13.494519 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.494382    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efb5861a-ce54-4654-b978-95e55325bdb5-maas-api-tls\") pod \"maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:13.494519 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.494431    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhgp\" (UniqueName: \"kubernetes.io/projected/efb5861a-ce54-4654-b978-95e55325bdb5-kube-api-access-vnhgp\") pod \"maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:13.595197 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.595165    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhgp\" (UniqueName: \"kubernetes.io/projected/efb5861a-ce54-4654-b978-95e55325bdb5-kube-api-access-vnhgp\") pod \"maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:13.595417 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.595263    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efb5861a-ce54-4654-b978-95e55325bdb5-openshift-service-ca\") pod \"maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:13.595417 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.595316    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efb5861a-ce54-4654-b978-95e55325bdb5-maas-api-tls\") pod \"maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:13.596042 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.596012    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efb5861a-ce54-4654-b978-95e55325bdb5-openshift-service-ca\") pod \"maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:13.597824 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.597802    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efb5861a-ce54-4654-b978-95e55325bdb5-maas-api-tls\") pod \"maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:13.603365 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.603292    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhgp\" (UniqueName: \"kubernetes.io/projected/efb5861a-ce54-4654-b978-95e55325bdb5-kube-api-access-vnhgp\") pod \"maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:13.666477 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.666392    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:13.818142 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:13.818112    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"]
Jul 25 01:46:13.818280 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:46:13.818242    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb5861a_ce54_4654_b978_95e55325bdb5.slice/crio-7d3e55a0cdac41cbf42185b9843d34d476163b9cb102f82a971b94d7e752089f WatchSource:0}: Error finding container 7d3e55a0cdac41cbf42185b9843d34d476163b9cb102f82a971b94d7e752089f: Status 404 returned error can't find the container with id 7d3e55a0cdac41cbf42185b9843d34d476163b9cb102f82a971b94d7e752089f
Jul 25 01:46:14.134577 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:14.134534    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74" event={"ID":"efb5861a-ce54-4654-b978-95e55325bdb5","Type":"ContainerStarted","Data":"7d3e55a0cdac41cbf42185b9843d34d476163b9cb102f82a971b94d7e752089f"}
Jul 25 01:46:15.141695 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:15.141656    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74" event={"ID":"efb5861a-ce54-4654-b978-95e55325bdb5","Type":"ContainerStarted","Data":"0911d6390840b175b7ee907980571fa17f95fd5b4afafff1d0779fe33b7ad9d1"}
Jul 25 01:46:15.142308 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:15.141751    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:15.158849 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:15.158803    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74" podStartSLOduration=1.584175485 podStartE2EDuration="2.158787575s" totalImagesPullingTime="574.61209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:46:15.157108696 +0000 UTC m=+2059.266917999" watchObservedRunningTime="2026-07-25 01:46:15.158787575 +0000 UTC m=+2059.268596853"
Jul 25 01:46:21.161860 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:21.161760    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:26.201558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:26.201525    2572 generic.go:356] "Generic (PLEG): container finished" podID="54519780-4c42-460d-a920-6229ce71cfeb" containerID="3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077" exitCode=0
Jul 25 01:46:26.201977 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:26.201603    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7" event={"ID":"54519780-4c42-460d-a920-6229ce71cfeb","Type":"ContainerDied","Data":"3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077"}
Jul 25 01:46:26.202024 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:26.201991    2572 scope.go:117] "RemoveContainer" containerID="3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077"
Jul 25 01:46:26.202948 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:26.202923    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:46:26.202998 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:26.202988    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:46:26.203032 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:26.203021    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:46:27.122458 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:27.122411    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:27.208892 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:27.208855    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7" event={"ID":"54519780-4c42-460d-a920-6229ce71cfeb","Type":"ContainerStarted","Data":"e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3"}
Jul 25 01:46:28.107645 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:28.107604    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:28.108959 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:28.108931    2572 patch_prober.go:28] interesting pod/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 25 01:46:28.109078 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:28.108991    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7" podUID="54519780-4c42-460d-a920-6229ce71cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 25 01:46:29.108384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:29.108342    2572 patch_prober.go:28] interesting pod/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 25 01:46:29.108765 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:29.108404    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7" podUID="54519780-4c42-460d-a920-6229ce71cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 25 01:46:30.108382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:30.108314    2572 patch_prober.go:28] interesting pod/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Jul 25 01:46:30.108558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:30.108404    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7" podUID="54519780-4c42-460d-a920-6229ce71cfeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Jul 25 01:46:31.112221 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:31.112189    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:31.112665 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:31.112569    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:31.113241 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:31.113223    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:35.118130 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.118096    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"]
Jul 25 01:46:35.118552 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.118370    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74" podUID="efb5861a-ce54-4654-b978-95e55325bdb5" containerName="maas-api" containerID="cri-o://0911d6390840b175b7ee907980571fa17f95fd5b4afafff1d0779fe33b7ad9d1" gracePeriod=30
Jul 25 01:46:35.250299 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.250268    2572 generic.go:356] "Generic (PLEG): container finished" podID="efb5861a-ce54-4654-b978-95e55325bdb5" containerID="0911d6390840b175b7ee907980571fa17f95fd5b4afafff1d0779fe33b7ad9d1" exitCode=0
Jul 25 01:46:35.250458 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.250355    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74" event={"ID":"efb5861a-ce54-4654-b978-95e55325bdb5","Type":"ContainerDied","Data":"0911d6390840b175b7ee907980571fa17f95fd5b4afafff1d0779fe33b7ad9d1"}
Jul 25 01:46:35.363719 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.363695    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:35.385566 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.385501    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efb5861a-ce54-4654-b978-95e55325bdb5-openshift-service-ca\") pod \"efb5861a-ce54-4654-b978-95e55325bdb5\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") "
Jul 25 01:46:35.385566 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.385559    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnhgp\" (UniqueName: \"kubernetes.io/projected/efb5861a-ce54-4654-b978-95e55325bdb5-kube-api-access-vnhgp\") pod \"efb5861a-ce54-4654-b978-95e55325bdb5\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") "
Jul 25 01:46:35.385725 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.385667    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efb5861a-ce54-4654-b978-95e55325bdb5-maas-api-tls\") pod \"efb5861a-ce54-4654-b978-95e55325bdb5\" (UID: \"efb5861a-ce54-4654-b978-95e55325bdb5\") "
Jul 25 01:46:35.385886 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.385863    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb5861a-ce54-4654-b978-95e55325bdb5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "efb5861a-ce54-4654-b978-95e55325bdb5" (UID: "efb5861a-ce54-4654-b978-95e55325bdb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:46:35.387647 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.387622    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb5861a-ce54-4654-b978-95e55325bdb5-kube-api-access-vnhgp" (OuterVolumeSpecName: "kube-api-access-vnhgp") pod "efb5861a-ce54-4654-b978-95e55325bdb5" (UID: "efb5861a-ce54-4654-b978-95e55325bdb5"). InnerVolumeSpecName "kube-api-access-vnhgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:46:35.387731 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.387625    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb5861a-ce54-4654-b978-95e55325bdb5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "efb5861a-ce54-4654-b978-95e55325bdb5" (UID: "efb5861a-ce54-4654-b978-95e55325bdb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:46:35.494351 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.494289    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnhgp\" (UniqueName: \"kubernetes.io/projected/efb5861a-ce54-4654-b978-95e55325bdb5-kube-api-access-vnhgp\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:35.494526 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.494358    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efb5861a-ce54-4654-b978-95e55325bdb5-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:35.494526 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:35.494374    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efb5861a-ce54-4654-b978-95e55325bdb5-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:36.256849 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:36.256824    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"
Jul 25 01:46:36.257395 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:36.256878    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74" event={"ID":"efb5861a-ce54-4654-b978-95e55325bdb5","Type":"ContainerDied","Data":"7d3e55a0cdac41cbf42185b9843d34d476163b9cb102f82a971b94d7e752089f"}
Jul 25 01:46:36.257395 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:36.256937    2572 scope.go:117] "RemoveContainer" containerID="0911d6390840b175b7ee907980571fa17f95fd5b4afafff1d0779fe33b7ad9d1"
Jul 25 01:46:36.282528 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:36.282504    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"]
Jul 25 01:46:36.288745 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:36.288718    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6d92f36d-854b8b4df-5ln74"]
Jul 25 01:46:36.353265 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:36.353230    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efb5861a-ce54-4654-b978-95e55325bdb5" path="/var/lib/kubelet/pods/efb5861a-ce54-4654-b978-95e55325bdb5/volumes"
Jul 25 01:46:46.458782 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:46.458745    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"]
Jul 25 01:46:46.459218 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:46.459001    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7" podUID="54519780-4c42-460d-a920-6229ce71cfeb" containerName="istio-proxy" containerID="cri-o://e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3" gracePeriod=30
Jul 25 01:46:51.697808 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.697783    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:51.839382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839261    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-socket\") pod \"54519780-4c42-460d-a920-6229ce71cfeb\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") "
Jul 25 01:46:51.839382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839346    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54519780-4c42-460d-a920-6229ce71cfeb-istio-podinfo\") pod \"54519780-4c42-460d-a920-6229ce71cfeb\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") "
Jul 25 01:46:51.839382 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839381    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-envoy\") pod \"54519780-4c42-460d-a920-6229ce71cfeb\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") "
Jul 25 01:46:51.839638 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839489    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-certs\") pod \"54519780-4c42-460d-a920-6229ce71cfeb\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") "
Jul 25 01:46:51.839638 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839530    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-istio-token\") pod \"54519780-4c42-460d-a920-6229ce71cfeb\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") "
Jul 25 01:46:51.839638 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839554    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-credential-socket\") pod \"54519780-4c42-460d-a920-6229ce71cfeb\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") "
Jul 25 01:46:51.839638 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839601    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "54519780-4c42-460d-a920-6229ce71cfeb" (UID: "54519780-4c42-460d-a920-6229ce71cfeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:46:51.839638 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839616    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54519780-4c42-460d-a920-6229ce71cfeb-istiod-ca-cert\") pod \"54519780-4c42-460d-a920-6229ce71cfeb\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") "
Jul 25 01:46:51.839860 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839681    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-data\") pod \"54519780-4c42-460d-a920-6229ce71cfeb\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") "
Jul 25 01:46:51.839860 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839726    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs6n5\" (UniqueName: \"kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-kube-api-access-hs6n5\") pod \"54519780-4c42-460d-a920-6229ce71cfeb\" (UID: \"54519780-4c42-460d-a920-6229ce71cfeb\") "
Jul 25 01:46:51.839860 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839793    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "54519780-4c42-460d-a920-6229ce71cfeb" (UID: "54519780-4c42-460d-a920-6229ce71cfeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:46:51.840025 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.839859    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "54519780-4c42-460d-a920-6229ce71cfeb" (UID: "54519780-4c42-460d-a920-6229ce71cfeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:46:51.840094 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.840017    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54519780-4c42-460d-a920-6229ce71cfeb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "54519780-4c42-460d-a920-6229ce71cfeb" (UID: "54519780-4c42-460d-a920-6229ce71cfeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:46:51.840094 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.840026    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:51.840094 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.840065    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-workload-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:51.840094 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.840091    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-credential-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:51.840969 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.840939    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-data" (OuterVolumeSpecName: "istio-data") pod "54519780-4c42-460d-a920-6229ce71cfeb" (UID: "54519780-4c42-460d-a920-6229ce71cfeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:46:51.841709 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.841689    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54519780-4c42-460d-a920-6229ce71cfeb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "54519780-4c42-460d-a920-6229ce71cfeb" (UID: "54519780-4c42-460d-a920-6229ce71cfeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 25 01:46:51.841796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.841726    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-istio-token" (OuterVolumeSpecName: "istio-token") pod "54519780-4c42-460d-a920-6229ce71cfeb" (UID: "54519780-4c42-460d-a920-6229ce71cfeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:46:51.841796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.841740    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "54519780-4c42-460d-a920-6229ce71cfeb" (UID: "54519780-4c42-460d-a920-6229ce71cfeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:46:51.841796 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.841784    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-kube-api-access-hs6n5" (OuterVolumeSpecName: "kube-api-access-hs6n5") pod "54519780-4c42-460d-a920-6229ce71cfeb" (UID: "54519780-4c42-460d-a920-6229ce71cfeb"). InnerVolumeSpecName "kube-api-access-hs6n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:46:51.941290 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.941254    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54519780-4c42-460d-a920-6229ce71cfeb-istiod-ca-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:51.941290 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.941287    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-data\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:51.941290 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.941296    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs6n5\" (UniqueName: \"kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-kube-api-access-hs6n5\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:51.941570 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.941305    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54519780-4c42-460d-a920-6229ce71cfeb-istio-podinfo\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:51.941570 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.941313    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54519780-4c42-460d-a920-6229ce71cfeb-istio-envoy\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:51.941570 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:51.941342    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54519780-4c42-460d-a920-6229ce71cfeb-istio-token\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:46:52.336192 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.336154    2572 generic.go:356] "Generic (PLEG): container finished" podID="54519780-4c42-460d-a920-6229ce71cfeb" containerID="e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3" exitCode=0
Jul 25 01:46:52.336392 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.336211    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"
Jul 25 01:46:52.336392 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.336241    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7" event={"ID":"54519780-4c42-460d-a920-6229ce71cfeb","Type":"ContainerDied","Data":"e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3"}
Jul 25 01:46:52.336392 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.336297    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7" event={"ID":"54519780-4c42-460d-a920-6229ce71cfeb","Type":"ContainerDied","Data":"80b4fce168ea1c0360573d44bb06a9926c70ab37446ad1af92f69738e52b5833"}
Jul 25 01:46:52.336392 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.336320    2572 scope.go:117] "RemoveContainer" containerID="e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3"
Jul 25 01:46:52.345637 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.345619    2572 scope.go:117] "RemoveContainer" containerID="3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077"
Jul 25 01:46:52.358227 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.357421    2572 scope.go:117] "RemoveContainer" containerID="e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3"
Jul 25 01:46:52.358595 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:46:52.358565    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3\": container with ID starting with e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3 not found: ID does not exist" containerID="e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3"
Jul 25 01:46:52.358648 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.358608    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3"} err="failed to get container status \"e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3\": rpc error: code = NotFound desc = could not find container \"e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3\": container with ID starting with e5d9d9b930af7cd76f4324f71fbd37dabe7860584e2314c1a2d6e28ef2253fd3 not found: ID does not exist"
Jul 25 01:46:52.358648 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.358636    2572 scope.go:117] "RemoveContainer" containerID="3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077"
Jul 25 01:46:52.359062 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:46:52.359035    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077\": container with ID starting with 3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077 not found: ID does not exist" containerID="3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077"
Jul 25 01:46:52.359206 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.359177    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077"} err="failed to get container status \"3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077\": rpc error: code = NotFound desc = could not find container \"3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077\": container with ID starting with 3947e053b736728bbaaab2208f99ef697a716ed4cb15bf34d04d527c9585e077 not found: ID does not exist"
Jul 25 01:46:52.360902 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.360881    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"]
Jul 25 01:46:52.362733 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:52.362712    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6d92f36d-openshift-default-895f7999b-xxvb7"]
Jul 25 01:46:54.352908 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:46:54.352874    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54519780-4c42-460d-a920-6229ce71cfeb" path="/var/lib/kubelet/pods/54519780-4c42-460d-a920-6229ce71cfeb/volumes"
Jul 25 01:47:06.119666 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.119627    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"]
Jul 25 01:47:06.120091 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.119928    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54519780-4c42-460d-a920-6229ce71cfeb" containerName="istio-proxy"
Jul 25 01:47:06.120091 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.119939    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="54519780-4c42-460d-a920-6229ce71cfeb" containerName="istio-proxy"
Jul 25 01:47:06.120091 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.119949    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb5861a-ce54-4654-b978-95e55325bdb5" containerName="maas-api"
Jul 25 01:47:06.120091 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.119954    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb5861a-ce54-4654-b978-95e55325bdb5" containerName="maas-api"
Jul 25 01:47:06.120091 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.120017    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb5861a-ce54-4654-b978-95e55325bdb5" containerName="maas-api"
Jul 25 01:47:06.120091 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.120026    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="54519780-4c42-460d-a920-6229ce71cfeb" containerName="istio-proxy"
Jul 25 01:47:06.120091 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.120034    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="54519780-4c42-460d-a920-6229ce71cfeb" containerName="istio-proxy"
Jul 25 01:47:06.123048 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.123027    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.125802 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.125780    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5e11b867-openshift-default-dockercfg-x4snq\""
Jul 25 01:47:06.125883 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.125784    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 25 01:47:06.138356 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.138298    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"]
Jul 25 01:47:06.258707 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.258669    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-credential-socket\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.258707 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.258711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-certs\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.258896 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.258732    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-token\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.258896 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.258818    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khjcr\" (UniqueName: \"kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-kube-api-access-khjcr\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.258896 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.258862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-data\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.258994 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.258892    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-envoy\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.258994 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.258927    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e270e82-65e8-4ee4-9351-de3aff96ad94-istiod-ca-cert\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.258994 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.258971    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-podinfo\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.259090 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.259008    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-socket\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.359742 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.359711    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-envoy\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.359890 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.359757    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e270e82-65e8-4ee4-9351-de3aff96ad94-istiod-ca-cert\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.359890 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.359784    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-podinfo\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.359890 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.359830    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-socket\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.360050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.359889    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-credential-socket\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.360050 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.359916    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-certs\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.360165 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.360053    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-token\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.360165 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.360121    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khjcr\" (UniqueName: \"kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-kube-api-access-khjcr\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.360165 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.360156    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-data\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.360345 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.360254    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-certs\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.360345 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.360299    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-socket\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.360729 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.360703    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-data\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.360838 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.360764    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-credential-socket\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.361088 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.361068    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e270e82-65e8-4ee4-9351-de3aff96ad94-istiod-ca-cert\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.362127 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.362099    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-envoy\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.362319 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.362300    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-podinfo\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.376708 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.376648    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khjcr\" (UniqueName: \"kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-kube-api-access-khjcr\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.379114 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.379093    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-token\") pod \"e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") " pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.433147 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.433119    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:06.568400 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.568373    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"]
Jul 25 01:47:06.570171 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:47:06.570144    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e270e82_65e8_4ee4_9351_de3aff96ad94.slice/crio-5b2774b55426b54e5320e30215555f3fbc4a13de78eb4c2a402f838254b59f87 WatchSource:0}: Error finding container 5b2774b55426b54e5320e30215555f3fbc4a13de78eb4c2a402f838254b59f87: Status 404 returned error can't find the container with id 5b2774b55426b54e5320e30215555f3fbc4a13de78eb4c2a402f838254b59f87
Jul 25 01:47:06.572264 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.572226    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:47:06.572392 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.572317    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:47:06.572392 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:06.572375    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:47:07.411727 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:07.411692    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" event={"ID":"4e270e82-65e8-4ee4-9351-de3aff96ad94","Type":"ContainerStarted","Data":"e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d"}
Jul 25 01:47:07.412194 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:07.411746    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" event={"ID":"4e270e82-65e8-4ee4-9351-de3aff96ad94","Type":"ContainerStarted","Data":"5b2774b55426b54e5320e30215555f3fbc4a13de78eb4c2a402f838254b59f87"}
Jul 25 01:47:07.433371 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:07.433340    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:07.435267 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:07.435227    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" podStartSLOduration=1.435210858 podStartE2EDuration="1.435210858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:47:07.43351535 +0000 UTC m=+2111.543324661" watchObservedRunningTime="2026-07-25 01:47:07.435210858 +0000 UTC m=+2111.545020210"
Jul 25 01:47:07.438406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:07.438384    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:08.417200 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:08.417163    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:08.418225 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:08.418204    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:09.515228 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.515193    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"]
Jul 25 01:47:09.515622 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.515507    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54519780-4c42-460d-a920-6229ce71cfeb" containerName="istio-proxy"
Jul 25 01:47:09.515622 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.515519    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="54519780-4c42-460d-a920-6229ce71cfeb" containerName="istio-proxy"
Jul 25 01:47:09.518494 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.518478    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:09.520930 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.520907    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhv77\""
Jul 25 01:47:09.521136 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.521120    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5e11b867-serving-cert\""
Jul 25 01:47:09.527861 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.527836    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"]
Jul 25 01:47:09.689765 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.689734    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00b04cdf-4a8c-4be4-bccd-a5a75f941969-maas-api-tls\") pod \"maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:09.689951 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.689781    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00b04cdf-4a8c-4be4-bccd-a5a75f941969-openshift-service-ca\") pod \"maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:09.689951 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.689877    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsfwf\" (UniqueName: \"kubernetes.io/projected/00b04cdf-4a8c-4be4-bccd-a5a75f941969-kube-api-access-rsfwf\") pod \"maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:09.790690 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.790592    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsfwf\" (UniqueName: \"kubernetes.io/projected/00b04cdf-4a8c-4be4-bccd-a5a75f941969-kube-api-access-rsfwf\") pod \"maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:09.790690 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.790651    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00b04cdf-4a8c-4be4-bccd-a5a75f941969-maas-api-tls\") pod \"maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:09.790690 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.790681    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00b04cdf-4a8c-4be4-bccd-a5a75f941969-openshift-service-ca\") pod \"maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:09.790980 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:47:09.790803    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-serving-cert: secret "maas-api-e2e-mt-5e11b867-serving-cert" not found
Jul 25 01:47:09.790980 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:47:09.790866    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00b04cdf-4a8c-4be4-bccd-a5a75f941969-maas-api-tls podName:00b04cdf-4a8c-4be4-bccd-a5a75f941969 nodeName:}" failed. No retries permitted until 2026-07-25 01:47:10.290851622 +0000 UTC m=+2114.400660883 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00b04cdf-4a8c-4be4-bccd-a5a75f941969-maas-api-tls") pod "maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf" (UID: "00b04cdf-4a8c-4be4-bccd-a5a75f941969") : secret "maas-api-e2e-mt-5e11b867-serving-cert" not found
Jul 25 01:47:09.791316 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.791297    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00b04cdf-4a8c-4be4-bccd-a5a75f941969-openshift-service-ca\") pod \"maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:09.804306 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:09.804287    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsfwf\" (UniqueName: \"kubernetes.io/projected/00b04cdf-4a8c-4be4-bccd-a5a75f941969-kube-api-access-rsfwf\") pod \"maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:10.296900 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:10.296857    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00b04cdf-4a8c-4be4-bccd-a5a75f941969-maas-api-tls\") pod \"maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:10.299428 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:10.299397    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00b04cdf-4a8c-4be4-bccd-a5a75f941969-maas-api-tls\") pod \"maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:10.429870 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:10.429840    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:10.766315 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:10.766289    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"]
Jul 25 01:47:10.768074 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:47:10.768036    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b04cdf_4a8c_4be4_bccd_a5a75f941969.slice/crio-ece2fc1381c87a03b7179b05026e22b4ca490ae27aaaf1aab40f04e7a2872630 WatchSource:0}: Error finding container ece2fc1381c87a03b7179b05026e22b4ca490ae27aaaf1aab40f04e7a2872630: Status 404 returned error can't find the container with id ece2fc1381c87a03b7179b05026e22b4ca490ae27aaaf1aab40f04e7a2872630
Jul 25 01:47:10.769744 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:10.769723    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 25 01:47:11.433264 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:11.433232    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf" event={"ID":"00b04cdf-4a8c-4be4-bccd-a5a75f941969","Type":"ContainerStarted","Data":"20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd"}
Jul 25 01:47:11.433470 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:11.433283    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf" event={"ID":"00b04cdf-4a8c-4be4-bccd-a5a75f941969","Type":"ContainerStarted","Data":"ece2fc1381c87a03b7179b05026e22b4ca490ae27aaaf1aab40f04e7a2872630"}
Jul 25 01:47:11.433470 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:11.433357    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:11.457735 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:11.457665    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf" podStartSLOduration=2.128233293 podStartE2EDuration="2.457648326s" totalImagesPullingTime="329.415033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:47:11.456752631 +0000 UTC m=+2115.566561942" watchObservedRunningTime="2026-07-25 01:47:11.457648326 +0000 UTC m=+2115.567457610"
Jul 25 01:47:17.443983 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:17.443950    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:47:21.997105 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:21.997067    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"]
Jul 25 01:47:22.000681 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.000663    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.003224 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.003201    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5e11b867\"/\"openshift-service-ca.crt\""
Jul 25 01:47:22.004136 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.004115    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5e11b867\"/\"default-dockercfg-rxwgs\""
Jul 25 01:47:22.004238 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.004115    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5e11b867\"/\"e2e-lifecycle-model-5e11b867-kserve-self-signed-certs\""
Jul 25 01:47:22.004238 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.004157    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5e11b867\"/\"kube-root-ca.crt\""
Jul 25 01:47:22.012001 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.011979    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"]
Jul 25 01:47:22.091770 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.091739    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-model-cache\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.091930 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.091806    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-dshm\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.091930 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.091838    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-home\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.091930 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.091895    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f532050-c297-4097-ab77-b71304fc3ab7-tls-certs\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.091930 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.091923    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-tmp-dir\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.092109 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.091946    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfrjl\" (UniqueName: \"kubernetes.io/projected/0f532050-c297-4097-ab77-b71304fc3ab7-kube-api-access-zfrjl\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.192572 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.192540    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-dshm\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.192726 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.192581    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-home\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.192726 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.192630    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f532050-c297-4097-ab77-b71304fc3ab7-tls-certs\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.192726 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.192656    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-tmp-dir\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.192726 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.192679    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfrjl\" (UniqueName: \"kubernetes.io/projected/0f532050-c297-4097-ab77-b71304fc3ab7-kube-api-access-zfrjl\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.192726 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.192715    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-model-cache\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.193070 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.193043    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-home\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.193156 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.193107    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-tmp-dir\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.193156 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.193144    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-model-cache\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.194790 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.194770    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-dshm\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.195162 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.195147    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f532050-c297-4097-ab77-b71304fc3ab7-tls-certs\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.209896 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.209868    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfrjl\" (UniqueName: \"kubernetes.io/projected/0f532050-c297-4097-ab77-b71304fc3ab7-kube-api-access-zfrjl\") pod \"e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") " pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.311739 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.311694    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:22.449027 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.449001    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"]
Jul 25 01:47:22.450903 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:47:22.450877    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f532050_c297_4097_ab77_b71304fc3ab7.slice/crio-cba8252255317bf10ada761f945e38464d5f5a7a5d4dd8d36958560e530bc484 WatchSource:0}: Error finding container cba8252255317bf10ada761f945e38464d5f5a7a5d4dd8d36958560e530bc484: Status 404 returned error can't find the container with id cba8252255317bf10ada761f945e38464d5f5a7a5d4dd8d36958560e530bc484
Jul 25 01:47:22.491387 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:22.491356    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h" event={"ID":"0f532050-c297-4097-ab77-b71304fc3ab7","Type":"ContainerStarted","Data":"cba8252255317bf10ada761f945e38464d5f5a7a5d4dd8d36958560e530bc484"}
Jul 25 01:47:23.499966 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:23.499929    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h" event={"ID":"0f532050-c297-4097-ab77-b71304fc3ab7","Type":"ContainerStarted","Data":"73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442"}
Jul 25 01:47:23.518304 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:23.518251    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h" podStartSLOduration=2.518235 podStartE2EDuration="2.518235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:47:23.516094304 +0000 UTC m=+2127.625903591" watchObservedRunningTime="2026-07-25 01:47:23.518235 +0000 UTC m=+2127.628044284"
Jul 25 01:47:24.458485 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:24.458445    2572 patch_prober.go:28] interesting pod/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 25 01:47:24.458672 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:24.458513    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 25 01:47:25.512722 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:25.512683    2572 generic.go:356] "Generic (PLEG): container finished" podID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerID="e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d" exitCode=0
Jul 25 01:47:25.513162 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:25.512759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" event={"ID":"4e270e82-65e8-4ee4-9351-de3aff96ad94","Type":"ContainerDied","Data":"e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d"}
Jul 25 01:47:25.513162 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:25.513135    2572 scope.go:117] "RemoveContainer" containerID="e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d"
Jul 25 01:47:25.514126 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:25.514099    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:47:25.514231 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:25.514177    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:47:25.514231 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:25.514223    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:47:26.519219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:26.519181    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" event={"ID":"4e270e82-65e8-4ee4-9351-de3aff96ad94","Type":"ContainerStarted","Data":"db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7"}
Jul 25 01:47:27.434076 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:27.434040    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:27.434249 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:27.434227    2572 patch_prober.go:28] interesting pod/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 25 01:47:27.434345 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:27.434273    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 25 01:47:28.434139 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:28.434099    2572 patch_prober.go:28] interesting pod/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 25 01:47:28.434584 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:28.434159    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 25 01:47:29.433889 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:29.433850    2572 patch_prober.go:28] interesting pod/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 25 01:47:29.434072 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:29.433912    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 25 01:47:30.436808 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:30.436776    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:30.437228 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:30.437153    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:30.437772 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:30.437753    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:47:32.311857 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:32.311770    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:32.311857 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:32.311825    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:32.324201 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:32.324171    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:32.560426 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:32.560387    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:47:56.566106 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:47:56.566073    2572 scope.go:117] "RemoveContainer" containerID="4c1488dcf4a5ba759b2bee4a45b231b787c200e3229ea5a49c1fb169de53ee79"
Jul 25 01:48:14.430525 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.430484    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"]
Jul 25 01:48:14.431023 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.430772    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf" podUID="00b04cdf-4a8c-4be4-bccd-a5a75f941969" containerName="maas-api" containerID="cri-o://20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd" gracePeriod=30
Jul 25 01:48:14.677450 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.677419    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:48:14.740164 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.740076    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00b04cdf-4a8c-4be4-bccd-a5a75f941969-openshift-service-ca\") pod \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") "
Jul 25 01:48:14.740313 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.740150    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00b04cdf-4a8c-4be4-bccd-a5a75f941969-maas-api-tls\") pod \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") "
Jul 25 01:48:14.740313 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.740203    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsfwf\" (UniqueName: \"kubernetes.io/projected/00b04cdf-4a8c-4be4-bccd-a5a75f941969-kube-api-access-rsfwf\") pod \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\" (UID: \"00b04cdf-4a8c-4be4-bccd-a5a75f941969\") "
Jul 25 01:48:14.740491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.740467    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b04cdf-4a8c-4be4-bccd-a5a75f941969-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00b04cdf-4a8c-4be4-bccd-a5a75f941969" (UID: "00b04cdf-4a8c-4be4-bccd-a5a75f941969"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:48:14.742353 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.742300    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00b04cdf-4a8c-4be4-bccd-a5a75f941969-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00b04cdf-4a8c-4be4-bccd-a5a75f941969" (UID: "00b04cdf-4a8c-4be4-bccd-a5a75f941969"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:48:14.742455 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.742368    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b04cdf-4a8c-4be4-bccd-a5a75f941969-kube-api-access-rsfwf" (OuterVolumeSpecName: "kube-api-access-rsfwf") pod "00b04cdf-4a8c-4be4-bccd-a5a75f941969" (UID: "00b04cdf-4a8c-4be4-bccd-a5a75f941969"). InnerVolumeSpecName "kube-api-access-rsfwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:48:14.761285 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.761258    2572 generic.go:356] "Generic (PLEG): container finished" podID="00b04cdf-4a8c-4be4-bccd-a5a75f941969" containerID="20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd" exitCode=0
Jul 25 01:48:14.761424 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.761300    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"
Jul 25 01:48:14.761424 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.761344    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf" event={"ID":"00b04cdf-4a8c-4be4-bccd-a5a75f941969","Type":"ContainerDied","Data":"20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd"}
Jul 25 01:48:14.761424 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.761393    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf" event={"ID":"00b04cdf-4a8c-4be4-bccd-a5a75f941969","Type":"ContainerDied","Data":"ece2fc1381c87a03b7179b05026e22b4ca490ae27aaaf1aab40f04e7a2872630"}
Jul 25 01:48:14.761424 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.761415    2572 scope.go:117] "RemoveContainer" containerID="20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd"
Jul 25 01:48:14.770159 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.770143    2572 scope.go:117] "RemoveContainer" containerID="20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd"
Jul 25 01:48:14.770448 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:48:14.770430    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd\": container with ID starting with 20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd not found: ID does not exist" containerID="20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd"
Jul 25 01:48:14.770511 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.770456    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd"} err="failed to get container status \"20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd\": rpc error: code = NotFound desc = could not find container \"20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd\": container with ID starting with 20c37cb29376896d0da8ef23f74c38778e61cb8a3b33afdd73b30368defe93fd not found: ID does not exist"
Jul 25 01:48:14.787785 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.787753    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"]
Jul 25 01:48:14.797709 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.797678    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5e11b867-54fdb88bb4-mprdf"]
Jul 25 01:48:14.841141 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.841104    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00b04cdf-4a8c-4be4-bccd-a5a75f941969-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:48:14.841300 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.841141    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsfwf\" (UniqueName: \"kubernetes.io/projected/00b04cdf-4a8c-4be4-bccd-a5a75f941969-kube-api-access-rsfwf\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:48:14.841300 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:14.841163    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00b04cdf-4a8c-4be4-bccd-a5a75f941969-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:48:16.352933 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:16.352898    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00b04cdf-4a8c-4be4-bccd-a5a75f941969" path="/var/lib/kubelet/pods/00b04cdf-4a8c-4be4-bccd-a5a75f941969/volumes"
Jul 25 01:48:27.473885 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:27.473849    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"]
Jul 25 01:48:42.493102 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.493036    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h" podUID="0f532050-c297-4097-ab77-b71304fc3ab7" containerName="main" containerID="cri-o://73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442" gracePeriod=45
Jul 25 01:48:42.550609 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.550569    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h" podUID="0f532050-c297-4097-ab77-b71304fc3ab7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 25 01:48:42.747401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.747309    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:48:42.890829 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.890789    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfrjl\" (UniqueName: \"kubernetes.io/projected/0f532050-c297-4097-ab77-b71304fc3ab7-kube-api-access-zfrjl\") pod \"0f532050-c297-4097-ab77-b71304fc3ab7\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") "
Jul 25 01:48:42.891007 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.890838    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-dshm\") pod \"0f532050-c297-4097-ab77-b71304fc3ab7\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") "
Jul 25 01:48:42.891007 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.890868    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-model-cache\") pod \"0f532050-c297-4097-ab77-b71304fc3ab7\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") "
Jul 25 01:48:42.891007 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.890925    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-tmp-dir\") pod \"0f532050-c297-4097-ab77-b71304fc3ab7\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") "
Jul 25 01:48:42.891007 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.890949    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-home\") pod \"0f532050-c297-4097-ab77-b71304fc3ab7\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") "
Jul 25 01:48:42.891007 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.891000    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f532050-c297-4097-ab77-b71304fc3ab7-tls-certs\") pod \"0f532050-c297-4097-ab77-b71304fc3ab7\" (UID: \"0f532050-c297-4097-ab77-b71304fc3ab7\") "
Jul 25 01:48:42.891258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.891162    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f532050-c297-4097-ab77-b71304fc3ab7" (UID: "0f532050-c297-4097-ab77-b71304fc3ab7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:48:42.891258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.891213    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-home" (OuterVolumeSpecName: "home") pod "0f532050-c297-4097-ab77-b71304fc3ab7" (UID: "0f532050-c297-4097-ab77-b71304fc3ab7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:48:42.891258 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.891232    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f532050-c297-4097-ab77-b71304fc3ab7" (UID: "0f532050-c297-4097-ab77-b71304fc3ab7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:48:42.891450 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.891273    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:48:42.891450 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.891291    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:48:42.893113 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.893089    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-dshm" (OuterVolumeSpecName: "dshm") pod "0f532050-c297-4097-ab77-b71304fc3ab7" (UID: "0f532050-c297-4097-ab77-b71304fc3ab7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:48:42.893226 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.893148    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f532050-c297-4097-ab77-b71304fc3ab7-kube-api-access-zfrjl" (OuterVolumeSpecName: "kube-api-access-zfrjl") pod "0f532050-c297-4097-ab77-b71304fc3ab7" (UID: "0f532050-c297-4097-ab77-b71304fc3ab7"). InnerVolumeSpecName "kube-api-access-zfrjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:48:42.893226 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.893163    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f532050-c297-4097-ab77-b71304fc3ab7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f532050-c297-4097-ab77-b71304fc3ab7" (UID: "0f532050-c297-4097-ab77-b71304fc3ab7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:48:42.903385 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.903361    2572 generic.go:356] "Generic (PLEG): container finished" podID="0f532050-c297-4097-ab77-b71304fc3ab7" containerID="73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442" exitCode=0
Jul 25 01:48:42.903505 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.903405    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"
Jul 25 01:48:42.903505 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.903440    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h" event={"ID":"0f532050-c297-4097-ab77-b71304fc3ab7","Type":"ContainerDied","Data":"73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442"}
Jul 25 01:48:42.903505 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.903480    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h" event={"ID":"0f532050-c297-4097-ab77-b71304fc3ab7","Type":"ContainerDied","Data":"cba8252255317bf10ada761f945e38464d5f5a7a5d4dd8d36958560e530bc484"}
Jul 25 01:48:42.903505 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.903497    2572 scope.go:117] "RemoveContainer" containerID="73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442"
Jul 25 01:48:42.912358 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.912315    2572 scope.go:117] "RemoveContainer" containerID="73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442"
Jul 25 01:48:42.912602 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:48:42.912586    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442\": container with ID starting with 73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442 not found: ID does not exist" containerID="73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442"
Jul 25 01:48:42.912652 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.912609    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442"} err="failed to get container status \"73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442\": rpc error: code = NotFound desc = could not find container \"73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442\": container with ID starting with 73ec3696faeeb8a542764250c730194d20d96b83d05867e580065b019e18a442 not found: ID does not exist"
Jul 25 01:48:42.924772 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.924744    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"]
Jul 25 01:48:42.928799 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.928778    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5e11b867/e2e-lifecycle-model-5e11b867-kserve-66bc56cc5-c976h"]
Jul 25 01:48:42.991870 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.991832    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfrjl\" (UniqueName: \"kubernetes.io/projected/0f532050-c297-4097-ab77-b71304fc3ab7-kube-api-access-zfrjl\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:48:42.991870 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.991863    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:48:42.991870 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.991878    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f532050-c297-4097-ab77-b71304fc3ab7-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:48:42.992114 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:42.991889    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f532050-c297-4097-ab77-b71304fc3ab7-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:48:44.352674 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:44.352640    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f532050-c297-4097-ab77-b71304fc3ab7" path="/var/lib/kubelet/pods/0f532050-c297-4097-ab77-b71304fc3ab7/volumes"
Jul 25 01:48:56.585572 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:48:56.585541    2572 scope.go:117] "RemoveContainer" containerID="de79f1ebd9231f87ec38af8b9992a70b86ff4e668ef0c3d2b9b24cf22a60b9b2"
Jul 25 01:49:16.259919 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:16.259880    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"]
Jul 25 01:49:16.260469 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:16.260150    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerName="istio-proxy" containerID="cri-o://db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7" gracePeriod=30
Jul 25 01:49:21.516667 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.516633    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:49:21.617911 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.617820    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-socket\") pod \"4e270e82-65e8-4ee4-9351-de3aff96ad94\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") "
Jul 25 01:49:21.617911 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.617897    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khjcr\" (UniqueName: \"kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-kube-api-access-khjcr\") pod \"4e270e82-65e8-4ee4-9351-de3aff96ad94\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") "
Jul 25 01:49:21.618150 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.617935    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e270e82-65e8-4ee4-9351-de3aff96ad94-istiod-ca-cert\") pod \"4e270e82-65e8-4ee4-9351-de3aff96ad94\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") "
Jul 25 01:49:21.618150 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.617976    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-podinfo\") pod \"4e270e82-65e8-4ee4-9351-de3aff96ad94\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") "
Jul 25 01:49:21.618150 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.618044    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-data\") pod \"4e270e82-65e8-4ee4-9351-de3aff96ad94\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") "
Jul 25 01:49:21.618150 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.618076    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-credential-socket\") pod \"4e270e82-65e8-4ee4-9351-de3aff96ad94\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") "
Jul 25 01:49:21.618150 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.618089    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e270e82-65e8-4ee4-9351-de3aff96ad94" (UID: "4e270e82-65e8-4ee4-9351-de3aff96ad94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:49:21.618150 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.618101    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-token\") pod \"4e270e82-65e8-4ee4-9351-de3aff96ad94\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") "
Jul 25 01:49:21.618491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.618163    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-envoy\") pod \"4e270e82-65e8-4ee4-9351-de3aff96ad94\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") "
Jul 25 01:49:21.618491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.618206    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-certs\") pod \"4e270e82-65e8-4ee4-9351-de3aff96ad94\" (UID: \"4e270e82-65e8-4ee4-9351-de3aff96ad94\") "
Jul 25 01:49:21.618591 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.618513    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:49:21.618725 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.618699    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e270e82-65e8-4ee4-9351-de3aff96ad94" (UID: "4e270e82-65e8-4ee4-9351-de3aff96ad94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:49:21.618980 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.618940    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e270e82-65e8-4ee4-9351-de3aff96ad94" (UID: "4e270e82-65e8-4ee4-9351-de3aff96ad94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:49:21.619090 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.618980    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e270e82-65e8-4ee4-9351-de3aff96ad94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e270e82-65e8-4ee4-9351-de3aff96ad94" (UID: "4e270e82-65e8-4ee4-9351-de3aff96ad94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:49:21.619374 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.619342    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e270e82-65e8-4ee4-9351-de3aff96ad94" (UID: "4e270e82-65e8-4ee4-9351-de3aff96ad94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:49:21.620499 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.620469    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e270e82-65e8-4ee4-9351-de3aff96ad94" (UID: "4e270e82-65e8-4ee4-9351-de3aff96ad94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:49:21.620701 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.620680    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-kube-api-access-khjcr" (OuterVolumeSpecName: "kube-api-access-khjcr") pod "4e270e82-65e8-4ee4-9351-de3aff96ad94" (UID: "4e270e82-65e8-4ee4-9351-de3aff96ad94"). InnerVolumeSpecName "kube-api-access-khjcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:49:21.620964 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.620942    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e270e82-65e8-4ee4-9351-de3aff96ad94" (UID: "4e270e82-65e8-4ee4-9351-de3aff96ad94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 25 01:49:21.621083 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.621055    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e270e82-65e8-4ee4-9351-de3aff96ad94" (UID: "4e270e82-65e8-4ee4-9351-de3aff96ad94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:49:21.719489 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.719448    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khjcr\" (UniqueName: \"kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-kube-api-access-khjcr\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:49:21.719489 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.719477    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e270e82-65e8-4ee4-9351-de3aff96ad94-istiod-ca-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:49:21.719489 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.719489    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-podinfo\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:49:21.719489 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.719497    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-data\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:49:21.719850 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.719505    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-credential-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:49:21.719850 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.719514    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-token\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:49:21.719850 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.719522    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-istio-envoy\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:49:21.719850 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:21.719529    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e270e82-65e8-4ee4-9351-de3aff96ad94-workload-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:49:22.113747 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.113706    2572 generic.go:356] "Generic (PLEG): container finished" podID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerID="db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7" exitCode=0
Jul 25 01:49:22.113909 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.113762    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"
Jul 25 01:49:22.113909 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.113772    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" event={"ID":"4e270e82-65e8-4ee4-9351-de3aff96ad94","Type":"ContainerDied","Data":"db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7"}
Jul 25 01:49:22.113909 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.113819    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v" event={"ID":"4e270e82-65e8-4ee4-9351-de3aff96ad94","Type":"ContainerDied","Data":"5b2774b55426b54e5320e30215555f3fbc4a13de78eb4c2a402f838254b59f87"}
Jul 25 01:49:22.113909 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.113843    2572 scope.go:117] "RemoveContainer" containerID="db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7"
Jul 25 01:49:22.123478 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.123461    2572 scope.go:117] "RemoveContainer" containerID="e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d"
Jul 25 01:49:22.131201 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.131184    2572 scope.go:117] "RemoveContainer" containerID="db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7"
Jul 25 01:49:22.131628 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:49:22.131608    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7\": container with ID starting with db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7 not found: ID does not exist" containerID="db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7"
Jul 25 01:49:22.131687 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.131636    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7"} err="failed to get container status \"db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7\": rpc error: code = NotFound desc = could not find container \"db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7\": container with ID starting with db3d4bc6fe2683323c6c5813858a384989d869c1b8cc967d2a0aff5d9acf25f7 not found: ID does not exist"
Jul 25 01:49:22.131687 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.131654    2572 scope.go:117] "RemoveContainer" containerID="e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d"
Jul 25 01:49:22.131909 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:49:22.131891    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d\": container with ID starting with e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d not found: ID does not exist" containerID="e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d"
Jul 25 01:49:22.131954 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.131915    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d"} err="failed to get container status \"e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d\": rpc error: code = NotFound desc = could not find container \"e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d\": container with ID starting with e3edc0e3f89fb7edd8413b27189ae60e34a6bf1563938166cb1d2f50fdf4411d not found: ID does not exist"
Jul 25 01:49:22.155147 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.155124    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"]
Jul 25 01:49:22.165163 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.165136    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5e11b867-openshift-default-6c669f76f6-2pn9v"]
Jul 25 01:49:22.353305 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:22.353269    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" path="/var/lib/kubelet/pods/4e270e82-65e8-4ee4-9351-de3aff96ad94/volumes"
Jul 25 01:49:56.613004 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:49:56.612969    2572 scope.go:117] "RemoveContainer" containerID="72b1bbde3537bf0e01b77d2993a86dcef9d3aa7ee596352099fbc3be6698001f"
Jul 25 01:50:11.920136 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920090    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"]
Jul 25 01:50:11.920681 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920661    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b04cdf-4a8c-4be4-bccd-a5a75f941969" containerName="maas-api"
Jul 25 01:50:11.920754 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920683    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b04cdf-4a8c-4be4-bccd-a5a75f941969" containerName="maas-api"
Jul 25 01:50:11.920754 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920700    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerName="istio-proxy"
Jul 25 01:50:11.920754 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920710    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerName="istio-proxy"
Jul 25 01:50:11.920754 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920722    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f532050-c297-4097-ab77-b71304fc3ab7" containerName="main"
Jul 25 01:50:11.920754 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920732    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f532050-c297-4097-ab77-b71304fc3ab7" containerName="main"
Jul 25 01:50:11.920754 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920752    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerName="istio-proxy"
Jul 25 01:50:11.921086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920763    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerName="istio-proxy"
Jul 25 01:50:11.921086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920840    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b04cdf-4a8c-4be4-bccd-a5a75f941969" containerName="maas-api"
Jul 25 01:50:11.921086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920853    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f532050-c297-4097-ab77-b71304fc3ab7" containerName="main"
Jul 25 01:50:11.921086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920869    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerName="istio-proxy"
Jul 25 01:50:11.921086 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.920878    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e270e82-65e8-4ee4-9351-de3aff96ad94" containerName="istio-proxy"
Jul 25 01:50:11.924246 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.924219    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:11.926853 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.926830    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 25 01:50:11.926853 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.926845    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-df5e5d-openshift-default-dockercfg-s74lz\""
Jul 25 01:50:11.939071 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:11.939040    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"]
Jul 25 01:50:12.030416 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.030374    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6f87\" (UniqueName: \"kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-kube-api-access-q6f87\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.030593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.030429    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-podinfo\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.030593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.030509    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-credential-socket\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.030593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.030547    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-data\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.030593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.030576    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-certs\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.030929 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.030655    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-socket\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.030929 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.030710    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-envoy\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.030929 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.030775    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-token\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.030929 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.030815    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istiod-ca-cert\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132200 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132163    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-podinfo\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132375 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132219    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-credential-socket\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132375 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132245    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-data\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132375 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132269    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-certs\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132375 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132300    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-socket\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132375 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132349    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-envoy\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132652 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132391    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-token\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132652 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132427    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istiod-ca-cert\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132652 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6f87\" (UniqueName: \"kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-kube-api-access-q6f87\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132827 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132705    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-certs\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132827 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132745    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-data\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.132983 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132841    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-credential-socket\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.133039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.132989    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-socket\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.133507 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.133477    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istiod-ca-cert\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.134917 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.134884    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-envoy\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.135025 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.135006    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-podinfo\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.140805 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.140780    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-token\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.142861 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.142838    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6f87\" (UniqueName: \"kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-kube-api-access-q6f87\") pod \"e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") " pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.238271 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.238187    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:12.377921 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.377896    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"]
Jul 25 01:50:12.380649 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:50:12.380622    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8116e67f_1b04_4c53_bbce_bacfaf8333c9.slice/crio-c69fc937ca45f8ab70428718566f55bd8fce1ece869b727695ddb2c14db68e6f WatchSource:0}: Error finding container c69fc937ca45f8ab70428718566f55bd8fce1ece869b727695ddb2c14db68e6f: Status 404 returned error can't find the container with id c69fc937ca45f8ab70428718566f55bd8fce1ece869b727695ddb2c14db68e6f
Jul 25 01:50:12.383043 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.383004    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:12.383126 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.383093    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:12.383177 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:12.383131    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:13.369181 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:13.369146    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" event={"ID":"8116e67f-1b04-4c53-bbce-bacfaf8333c9","Type":"ContainerStarted","Data":"e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4"}
Jul 25 01:50:13.369605 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:13.369193    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" event={"ID":"8116e67f-1b04-4c53-bbce-bacfaf8333c9","Type":"ContainerStarted","Data":"c69fc937ca45f8ab70428718566f55bd8fce1ece869b727695ddb2c14db68e6f"}
Jul 25 01:50:13.392996 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:13.392944    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" podStartSLOduration=2.392930342 podStartE2EDuration="2.392930342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:50:13.39136414 +0000 UTC m=+2297.501173420" watchObservedRunningTime="2026-07-25 01:50:13.392930342 +0000 UTC m=+2297.502739625"
Jul 25 01:50:14.239109 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:14.239072    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:14.244546 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:14.244520    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:14.374135 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:14.374099    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:14.375043 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:14.375024    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:15.430907 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.430874    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"]
Jul 25 01:50:15.434819 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.434798    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:15.437142 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.437115    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-df5e5d-serving-cert\""
Jul 25 01:50:15.437279 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.437231    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhv77\""
Jul 25 01:50:15.444280 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.444256    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"]
Jul 25 01:50:15.564420 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.564381    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ppzq\" (UniqueName: \"kubernetes.io/projected/3054967b-e564-4455-8c72-7f0bea819952-kube-api-access-4ppzq\") pod \"maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:15.564609 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.564438    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3054967b-e564-4455-8c72-7f0bea819952-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:15.564609 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.564502    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3054967b-e564-4455-8c72-7f0bea819952-maas-api-tls\") pod \"maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:15.665882 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.665842    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3054967b-e564-4455-8c72-7f0bea819952-maas-api-tls\") pod \"maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:15.666049 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.665916    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppzq\" (UniqueName: \"kubernetes.io/projected/3054967b-e564-4455-8c72-7f0bea819952-kube-api-access-4ppzq\") pod \"maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:15.666049 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.665947    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3054967b-e564-4455-8c72-7f0bea819952-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:15.666049 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:50:15.666023    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-serving-cert: secret "maas-api-e2e-inf-a-df5e5d-serving-cert" not found
Jul 25 01:50:15.666187 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:50:15.666101    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3054967b-e564-4455-8c72-7f0bea819952-maas-api-tls podName:3054967b-e564-4455-8c72-7f0bea819952 nodeName:}" failed. No retries permitted until 2026-07-25 01:50:16.166079908 +0000 UTC m=+2300.275889173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3054967b-e564-4455-8c72-7f0bea819952-maas-api-tls") pod "maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj" (UID: "3054967b-e564-4455-8c72-7f0bea819952") : secret "maas-api-e2e-inf-a-df5e5d-serving-cert" not found
Jul 25 01:50:15.666624 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.666606    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3054967b-e564-4455-8c72-7f0bea819952-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:15.675242 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:15.675220    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppzq\" (UniqueName: \"kubernetes.io/projected/3054967b-e564-4455-8c72-7f0bea819952-kube-api-access-4ppzq\") pod \"maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:16.171077 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:16.171037    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3054967b-e564-4455-8c72-7f0bea819952-maas-api-tls\") pod \"maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:16.173555 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:16.173533    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3054967b-e564-4455-8c72-7f0bea819952-maas-api-tls\") pod \"maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:16.347162 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:16.347124    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:16.483820 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:16.483781    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"]
Jul 25 01:50:16.485118 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:50:16.485091    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3054967b_e564_4455_8c72_7f0bea819952.slice/crio-222b3a65974995da0ecd8ef0a71d1300318843c89e62323798f7801908937876 WatchSource:0}: Error finding container 222b3a65974995da0ecd8ef0a71d1300318843c89e62323798f7801908937876: Status 404 returned error can't find the container with id 222b3a65974995da0ecd8ef0a71d1300318843c89e62323798f7801908937876
Jul 25 01:50:17.389967 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:17.389886    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj" event={"ID":"3054967b-e564-4455-8c72-7f0bea819952","Type":"ContainerStarted","Data":"3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e"}
Jul 25 01:50:17.389967 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:17.389939    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj" event={"ID":"3054967b-e564-4455-8c72-7f0bea819952","Type":"ContainerStarted","Data":"222b3a65974995da0ecd8ef0a71d1300318843c89e62323798f7801908937876"}
Jul 25 01:50:17.389967 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:17.389968    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:17.409426 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:17.409372    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj" podStartSLOduration=1.969553523 podStartE2EDuration="2.409356119s" totalImagesPullingTime="439.802596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:50:17.407739782 +0000 UTC m=+2301.517549064" watchObservedRunningTime="2026-07-25 01:50:17.409356119 +0000 UTC m=+2301.519165402"
Jul 25 01:50:20.547587 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.547546    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"]
Jul 25 01:50:20.551295 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.551269    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.553882 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.553860    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-320e6c-openshift-default-dockercfg-tb44d\""
Jul 25 01:50:20.567547 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.567521    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"]
Jul 25 01:50:20.704678 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.704637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-socket\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.704678 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.704680    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-credential-socket\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.704940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.704700    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-podinfo\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.704940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.704796    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6lcw\" (UniqueName: \"kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-kube-api-access-s6lcw\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.704940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.704847    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-token\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.704940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.704877    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istiod-ca-cert\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.704940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.704933    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-certs\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.705208 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.704949    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-data\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.705208 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.704967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-envoy\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.805792 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.805699    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6lcw\" (UniqueName: \"kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-kube-api-access-s6lcw\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.805792 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.805754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-token\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806014 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.805795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istiod-ca-cert\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806014 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.805871    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-certs\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806014 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.805888    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-data\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806014 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.805907    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-envoy\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806014 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.805938    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-socket\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806014 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.805965    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-credential-socket\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806014 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.805988    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-podinfo\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.806282    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-data\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.806348    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-certs\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806518 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.806421    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-socket\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806570 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.806493    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-credential-socket\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.806769 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.806750    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istiod-ca-cert\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.808285 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.808259    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-envoy\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.808520 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.808498    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-podinfo\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.813783 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.813759    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-token\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.814127 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.814110    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6lcw\" (UniqueName: \"kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-kube-api-access-s6lcw\") pod \"e2e-inf-b-320e6c-openshift-default-6769647446-khs4t\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") " pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.861861 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.861837    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:20.995220 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:20.995189    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"]
Jul 25 01:50:20.997821 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:50:20.997796    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72f9283_1fe4_43b7_9614_fc1a0f4e7964.slice/crio-9c077fa099a00a40a23e37c7f39405afdf0d74fd5c74e85ab0220493718572fa WatchSource:0}: Error finding container 9c077fa099a00a40a23e37c7f39405afdf0d74fd5c74e85ab0220493718572fa: Status 404 returned error can't find the container with id 9c077fa099a00a40a23e37c7f39405afdf0d74fd5c74e85ab0220493718572fa
Jul 25 01:50:21.000501 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:21.000471    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:21.000578 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:21.000532    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:21.000578 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:21.000560    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:21.411918 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:21.411887    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" event={"ID":"e72f9283-1fe4-43b7-9614-fc1a0f4e7964","Type":"ContainerStarted","Data":"874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307"}
Jul 25 01:50:21.412089 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:21.411934    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" event={"ID":"e72f9283-1fe4-43b7-9614-fc1a0f4e7964","Type":"ContainerStarted","Data":"9c077fa099a00a40a23e37c7f39405afdf0d74fd5c74e85ab0220493718572fa"}
Jul 25 01:50:21.434571 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:21.434520    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" podStartSLOduration=1.434505941 podStartE2EDuration="1.434505941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:50:21.432880083 +0000 UTC m=+2305.542689405" watchObservedRunningTime="2026-07-25 01:50:21.434505941 +0000 UTC m=+2305.544315225"
Jul 25 01:50:21.862596 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:21.862521    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:22.868225 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:22.868189    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:23.400485 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.400456    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:50:23.422357 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.422175    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:23.423270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.423243    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:23.621581 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.621544    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"]
Jul 25 01:50:23.625644 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.625625    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:23.628264 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.628240    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-320e6c-serving-cert\""
Jul 25 01:50:23.633601 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.633542    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"]
Jul 25 01:50:23.730285 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.730199    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb69d\" (UniqueName: \"kubernetes.io/projected/f25828b2-87fd-4194-a872-e179cc80f599-kube-api-access-sb69d\") pod \"maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:23.730285 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.730237    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25828b2-87fd-4194-a872-e179cc80f599-openshift-service-ca\") pod \"maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:23.730535 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.730304    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f25828b2-87fd-4194-a872-e179cc80f599-maas-api-tls\") pod \"maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:23.831496 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.831459    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f25828b2-87fd-4194-a872-e179cc80f599-maas-api-tls\") pod \"maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:23.831644 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.831573    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb69d\" (UniqueName: \"kubernetes.io/projected/f25828b2-87fd-4194-a872-e179cc80f599-kube-api-access-sb69d\") pod \"maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:23.831644 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:50:23.831604    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-serving-cert: secret "maas-api-e2e-inf-b-320e6c-serving-cert" not found
Jul 25 01:50:23.831791 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.831606    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25828b2-87fd-4194-a872-e179cc80f599-openshift-service-ca\") pod \"maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:23.831791 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:50:23.831680    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f25828b2-87fd-4194-a872-e179cc80f599-maas-api-tls podName:f25828b2-87fd-4194-a872-e179cc80f599 nodeName:}" failed. No retries permitted until 2026-07-25 01:50:24.331657601 +0000 UTC m=+2308.441466871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f25828b2-87fd-4194-a872-e179cc80f599-maas-api-tls") pod "maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk" (UID: "f25828b2-87fd-4194-a872-e179cc80f599") : secret "maas-api-e2e-inf-b-320e6c-serving-cert" not found
Jul 25 01:50:23.832176 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.832157    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25828b2-87fd-4194-a872-e179cc80f599-openshift-service-ca\") pod \"maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:23.846903 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:23.846847    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb69d\" (UniqueName: \"kubernetes.io/projected/f25828b2-87fd-4194-a872-e179cc80f599-kube-api-access-sb69d\") pod \"maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:24.336010 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:24.335969    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f25828b2-87fd-4194-a872-e179cc80f599-maas-api-tls\") pod \"maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:24.351346 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:24.348873    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f25828b2-87fd-4194-a872-e179cc80f599-maas-api-tls\") pod \"maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:24.559344 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:24.556480    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:24.720876 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:24.720846    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"]
Jul 25 01:50:24.722408 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:50:24.722382    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25828b2_87fd_4194_a872_e179cc80f599.slice/crio-b00782c9eabaed7fab7abeca0443f06f8dfd928015d2d1a3cfe7df1285801069 WatchSource:0}: Error finding container b00782c9eabaed7fab7abeca0443f06f8dfd928015d2d1a3cfe7df1285801069: Status 404 returned error can't find the container with id b00782c9eabaed7fab7abeca0443f06f8dfd928015d2d1a3cfe7df1285801069
Jul 25 01:50:25.433413 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:25.433376    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk" event={"ID":"f25828b2-87fd-4194-a872-e179cc80f599","Type":"ContainerStarted","Data":"37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5"}
Jul 25 01:50:25.433809 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:25.433431    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk" event={"ID":"f25828b2-87fd-4194-a872-e179cc80f599","Type":"ContainerStarted","Data":"b00782c9eabaed7fab7abeca0443f06f8dfd928015d2d1a3cfe7df1285801069"}
Jul 25 01:50:25.433809 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:25.433477    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:25.450458 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:25.450379    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk" podStartSLOduration=2.100296712 podStartE2EDuration="2.450358673s" totalImagesPullingTime="350.061961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:50:25.449042845 +0000 UTC m=+2309.558852156" watchObservedRunningTime="2026-07-25 01:50:25.450358673 +0000 UTC m=+2309.560167957"
Jul 25 01:50:29.375766 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:29.375681    2572 patch_prober.go:28] interesting pod/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 25 01:50:29.375766 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:29.375737    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 25 01:50:29.462288 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:29.462181    2572 generic.go:356] "Generic (PLEG): container finished" podID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerID="e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4" exitCode=0
Jul 25 01:50:29.462288 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:29.462243    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" event={"ID":"8116e67f-1b04-4c53-bbce-bacfaf8333c9","Type":"ContainerDied","Data":"e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4"}
Jul 25 01:50:29.462792 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:29.462766    2572 scope.go:117] "RemoveContainer" containerID="e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4"
Jul 25 01:50:29.464088 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:29.464049    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:29.464198 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:29.464138    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:29.464198 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:29.464177    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:30.473738 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:30.473692    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" event={"ID":"8116e67f-1b04-4c53-bbce-bacfaf8333c9","Type":"ContainerStarted","Data":"7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e"}
Jul 25 01:50:31.239685 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:31.239228    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:31.239685 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:31.239383    2572 patch_prober.go:28] interesting pod/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 25 01:50:31.239685 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:31.239431    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 25 01:50:31.451692 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:31.451644    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:50:32.239376 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:32.239260    2572 patch_prober.go:28] interesting pod/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 25 01:50:32.239947 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:32.239914    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 25 01:50:33.239579 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:33.239535    2572 patch_prober.go:28] interesting pod/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 25 01:50:33.240074 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:33.239612    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 25 01:50:34.239539 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:34.239498    2572 patch_prober.go:28] interesting pod/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 25 01:50:34.239821 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:34.239792    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 25 01:50:35.242005 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:35.241921    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:35.242522 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:35.242311    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:35.242953 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:35.242936    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:50:35.502174 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:35.502093    2572 generic.go:356] "Generic (PLEG): container finished" podID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerID="874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307" exitCode=0
Jul 25 01:50:35.502340 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:35.502167    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" event={"ID":"e72f9283-1fe4-43b7-9614-fc1a0f4e7964","Type":"ContainerDied","Data":"874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307"}
Jul 25 01:50:35.502665 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:35.502648    2572 scope.go:117] "RemoveContainer" containerID="874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307"
Jul 25 01:50:35.503617 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:35.503593    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:35.503676 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:35.503658    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:35.503725 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:35.503688    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:50:36.509434 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:36.509398    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" event={"ID":"e72f9283-1fe4-43b7-9614-fc1a0f4e7964","Type":"ContainerStarted","Data":"15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8"}
Jul 25 01:50:36.862558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:36.862459    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:36.863743 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:36.863710    2572 patch_prober.go:28] interesting pod/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 25 01:50:36.863872 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:36.863776    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 25 01:50:37.862520 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:37.862478    2572 patch_prober.go:28] interesting pod/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 25 01:50:37.863030 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:37.862534    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 25 01:50:38.422812 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:38.422782    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:38.862745 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:38.862648    2572 patch_prober.go:28] interesting pod/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 25 01:50:38.862745 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:38.862726    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 25 01:50:39.862629 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:39.862588    2572 patch_prober.go:28] interesting pod/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Jul 25 01:50:39.862815 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:39.862650    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Jul 25 01:50:40.692462 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.692424    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"]
Jul 25 01:50:40.694854 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.694837    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.697473 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.697452    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-df5e5d\"/\"kube-root-ca.crt\""
Jul 25 01:50:40.698390 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.698367    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-df5e5d\"/\"test-model-df5e5d-kserve-self-signed-certs\""
Jul 25 01:50:40.698507 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.698407    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-df5e5d\"/\"openshift-service-ca.crt\""
Jul 25 01:50:40.698507 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.698407    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-df5e5d\"/\"default-dockercfg-cf8xj\""
Jul 25 01:50:40.707133 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.707112    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"]
Jul 25 01:50:40.789673 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.789632    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tmp-dir\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.789673 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.789674    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-dshm\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.789905 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.789696    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-model-cache\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.789905 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.789792    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tls-certs\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.789905 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.789827    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g4p2\" (UniqueName: \"kubernetes.io/projected/c955564e-6a8b-4b18-b4b3-68ed745a30d7-kube-api-access-7g4p2\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.789905 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.789861    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-home\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.866790 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.866759    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:40.867731 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.867711    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:50:40.890766 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.890738    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-home\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.890888 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.890781    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tmp-dir\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.890934 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.890919    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-dshm\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.891030 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.890960    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-model-cache\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.891108 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.891082    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tls-certs\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.891171 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.891134    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g4p2\" (UniqueName: \"kubernetes.io/projected/c955564e-6a8b-4b18-b4b3-68ed745a30d7-kube-api-access-7g4p2\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.891231 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.891166    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-home\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.891231 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.891193    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tmp-dir\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.891373 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.891303    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-model-cache\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.893152 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.893127    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-dshm\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.893448 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.893431    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tls-certs\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:40.898507 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:40.898489    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g4p2\" (UniqueName: \"kubernetes.io/projected/c955564e-6a8b-4b18-b4b3-68ed745a30d7-kube-api-access-7g4p2\") pod \"test-model-df5e5d-kserve-79cb686f9f-w7brx\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") " pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:41.004128 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:41.004037    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:41.137100 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:41.137074    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"]
Jul 25 01:50:41.138409 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:50:41.138379    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc955564e_6a8b_4b18_b4b3_68ed745a30d7.slice/crio-054a9008dc6f23a0b7b2e1c163cb4220b1a527fc5ae5ff31300a7533f8195f79 WatchSource:0}: Error finding container 054a9008dc6f23a0b7b2e1c163cb4220b1a527fc5ae5ff31300a7533f8195f79: Status 404 returned error can't find the container with id 054a9008dc6f23a0b7b2e1c163cb4220b1a527fc5ae5ff31300a7533f8195f79
Jul 25 01:50:41.539166 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:41.539129    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx" event={"ID":"c955564e-6a8b-4b18-b4b3-68ed745a30d7","Type":"ContainerStarted","Data":"edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c"}
Jul 25 01:50:41.539166 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:41.539173    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx" event={"ID":"c955564e-6a8b-4b18-b4b3-68ed745a30d7","Type":"ContainerStarted","Data":"054a9008dc6f23a0b7b2e1c163cb4220b1a527fc5ae5ff31300a7533f8195f79"}
Jul 25 01:50:41.556451 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:41.556394    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx" podStartSLOduration=1.556377453 podStartE2EDuration="1.556377453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:50:41.555717347 +0000 UTC m=+2325.665526632" watchObservedRunningTime="2026-07-25 01:50:41.556377453 +0000 UTC m=+2325.666186737"
Jul 25 01:50:51.004671 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:51.004632    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:51.005099 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:51.004722    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:51.017838 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:51.017815    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:50:51.615491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:50:51.615460    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:51:19.200586 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.200550    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"]
Jul 25 01:51:19.204173 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.204155    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.207279 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.207257    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-320e6c\"/\"test-model-320e6c-kserve-self-signed-certs\""
Jul 25 01:51:19.207393 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.207260    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-320e6c\"/\"openshift-service-ca.crt\""
Jul 25 01:51:19.207393 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.207258    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-320e6c\"/\"default-dockercfg-s4s76\""
Jul 25 01:51:19.207393 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.207257    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-320e6c\"/\"kube-root-ca.crt\""
Jul 25 01:51:19.212738 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.212715    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"]
Jul 25 01:51:19.319127 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.319091    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-model-cache\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.319295 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.319136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-tmp-dir\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.319295 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.319166    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f6711-c45a-4458-8e06-16bcf3694b93-tls-certs\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.319295 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.319252    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcx95\" (UniqueName: \"kubernetes.io/projected/aa4f6711-c45a-4458-8e06-16bcf3694b93-kube-api-access-jcx95\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.319295 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.319287    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-dshm\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.319469 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.319384    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-home\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.420202 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.420159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f6711-c45a-4458-8e06-16bcf3694b93-tls-certs\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.420406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.420212    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcx95\" (UniqueName: \"kubernetes.io/projected/aa4f6711-c45a-4458-8e06-16bcf3694b93-kube-api-access-jcx95\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.420406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.420235    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-dshm\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.420406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.420271    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-home\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.420406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.420297    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-model-cache\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.420406 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.420335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-tmp-dir\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.420775 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.420751    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-home\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.420903 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.420831    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-model-cache\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.420967 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.420936    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-tmp-dir\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.422730 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.422702    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-dshm\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.423170 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.423150    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f6711-c45a-4458-8e06-16bcf3694b93-tls-certs\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.428179 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.428153    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcx95\" (UniqueName: \"kubernetes.io/projected/aa4f6711-c45a-4458-8e06-16bcf3694b93-kube-api-access-jcx95\") pod \"test-model-320e6c-kserve-784dfdd5ff-j856v\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") " pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.514202 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.514122    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:19.645953 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.645919    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"]
Jul 25 01:51:19.647294 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:51:19.647268    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa4f6711_c45a_4458_8e06_16bcf3694b93.slice/crio-48e841fb73c56997ef7d521e913a4b6b9ce335d7b19d272cd75e4448e0c91383 WatchSource:0}: Error finding container 48e841fb73c56997ef7d521e913a4b6b9ce335d7b19d272cd75e4448e0c91383: Status 404 returned error can't find the container with id 48e841fb73c56997ef7d521e913a4b6b9ce335d7b19d272cd75e4448e0c91383
Jul 25 01:51:19.735524 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.735491    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v" event={"ID":"aa4f6711-c45a-4458-8e06-16bcf3694b93","Type":"ContainerStarted","Data":"bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c"}
Jul 25 01:51:19.735668 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.735539    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v" event={"ID":"aa4f6711-c45a-4458-8e06-16bcf3694b93","Type":"ContainerStarted","Data":"48e841fb73c56997ef7d521e913a4b6b9ce335d7b19d272cd75e4448e0c91383"}
Jul 25 01:51:19.755849 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:19.755803    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v" podStartSLOduration=0.755787381 podStartE2EDuration="755.787381ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:51:19.75448767 +0000 UTC m=+2363.864296954" watchObservedRunningTime="2026-07-25 01:51:19.755787381 +0000 UTC m=+2363.865596713"
Jul 25 01:51:29.514410 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:29.514369    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:29.514802 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:29.514428    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:29.527230 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:29.527199    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:51:29.799990 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:51:29.799905    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:52:03.027734 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:03.027700    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"]
Jul 25 01:52:04.027970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:04.027939    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"]
Jul 25 01:52:14.531139 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:14.531102    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"]
Jul 25 01:52:14.531639 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:14.531309    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj" podUID="3054967b-e564-4455-8c72-7f0bea819952" containerName="maas-api" containerID="cri-o://3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e" gracePeriod=30
Jul 25 01:52:14.986789 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:14.986764    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:52:15.016769 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.016743    2572 generic.go:356] "Generic (PLEG): container finished" podID="3054967b-e564-4455-8c72-7f0bea819952" containerID="3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e" exitCode=0
Jul 25 01:52:15.016934 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.016795    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"
Jul 25 01:52:15.016934 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.016826    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj" event={"ID":"3054967b-e564-4455-8c72-7f0bea819952","Type":"ContainerDied","Data":"3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e"}
Jul 25 01:52:15.016934 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.016904    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj" event={"ID":"3054967b-e564-4455-8c72-7f0bea819952","Type":"ContainerDied","Data":"222b3a65974995da0ecd8ef0a71d1300318843c89e62323798f7801908937876"}
Jul 25 01:52:15.016934 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.016934    2572 scope.go:117] "RemoveContainer" containerID="3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e"
Jul 25 01:52:15.026928 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.026906    2572 scope.go:117] "RemoveContainer" containerID="3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e"
Jul 25 01:52:15.027199 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:52:15.027180    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e\": container with ID starting with 3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e not found: ID does not exist" containerID="3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e"
Jul 25 01:52:15.027274 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.027223    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e"} err="failed to get container status \"3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e\": rpc error: code = NotFound desc = could not find container \"3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e\": container with ID starting with 3158e42b5d38c9215188e437b777191440d1fe4903f332d2f1370fd1ca0db26e not found: ID does not exist"
Jul 25 01:52:15.104179 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.104092    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3054967b-e564-4455-8c72-7f0bea819952-maas-api-tls\") pod \"3054967b-e564-4455-8c72-7f0bea819952\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") "
Jul 25 01:52:15.104179 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.104175    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ppzq\" (UniqueName: \"kubernetes.io/projected/3054967b-e564-4455-8c72-7f0bea819952-kube-api-access-4ppzq\") pod \"3054967b-e564-4455-8c72-7f0bea819952\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") "
Jul 25 01:52:15.104432 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.104206    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3054967b-e564-4455-8c72-7f0bea819952-openshift-service-ca\") pod \"3054967b-e564-4455-8c72-7f0bea819952\" (UID: \"3054967b-e564-4455-8c72-7f0bea819952\") "
Jul 25 01:52:15.104615 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.104590    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3054967b-e564-4455-8c72-7f0bea819952-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3054967b-e564-4455-8c72-7f0bea819952" (UID: "3054967b-e564-4455-8c72-7f0bea819952"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:52:15.106311 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.106286    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3054967b-e564-4455-8c72-7f0bea819952-kube-api-access-4ppzq" (OuterVolumeSpecName: "kube-api-access-4ppzq") pod "3054967b-e564-4455-8c72-7f0bea819952" (UID: "3054967b-e564-4455-8c72-7f0bea819952"). InnerVolumeSpecName "kube-api-access-4ppzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:52:15.106468 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.106336    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3054967b-e564-4455-8c72-7f0bea819952-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3054967b-e564-4455-8c72-7f0bea819952" (UID: "3054967b-e564-4455-8c72-7f0bea819952"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:52:15.205706 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.205664    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ppzq\" (UniqueName: \"kubernetes.io/projected/3054967b-e564-4455-8c72-7f0bea819952-kube-api-access-4ppzq\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:15.205706 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.205701    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3054967b-e564-4455-8c72-7f0bea819952-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:15.205706 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.205712    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3054967b-e564-4455-8c72-7f0bea819952-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:15.337466 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.337436    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"]
Jul 25 01:52:15.340906 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:15.340881    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df5e5d-6c6b8d7646-x69rj"]
Jul 25 01:52:16.354534 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:16.354501    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3054967b-e564-4455-8c72-7f0bea819952" path="/var/lib/kubelet/pods/3054967b-e564-4455-8c72-7f0bea819952/volumes"
Jul 25 01:52:18.048615 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.048554    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx" podUID="c955564e-6a8b-4b18-b4b3-68ed745a30d7" containerName="main" containerID="cri-o://edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c" gracePeriod=45
Jul 25 01:52:18.299397 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.299341    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:52:18.433198 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.433169    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-dshm\") pod \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") "
Jul 25 01:52:18.433381 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.433220    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-model-cache\") pod \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") "
Jul 25 01:52:18.433381 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.433285    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tls-certs\") pod \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") "
Jul 25 01:52:18.433381 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.433303    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g4p2\" (UniqueName: \"kubernetes.io/projected/c955564e-6a8b-4b18-b4b3-68ed745a30d7-kube-api-access-7g4p2\") pod \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") "
Jul 25 01:52:18.433381 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.433364    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tmp-dir\") pod \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") "
Jul 25 01:52:18.433607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.433387    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-home\") pod \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\" (UID: \"c955564e-6a8b-4b18-b4b3-68ed745a30d7\") "
Jul 25 01:52:18.433607 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.433551    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "c955564e-6a8b-4b18-b4b3-68ed745a30d7" (UID: "c955564e-6a8b-4b18-b4b3-68ed745a30d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:18.433717 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.433683    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c955564e-6a8b-4b18-b4b3-68ed745a30d7" (UID: "c955564e-6a8b-4b18-b4b3-68ed745a30d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:18.433780 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.433718    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-home" (OuterVolumeSpecName: "home") pod "c955564e-6a8b-4b18-b4b3-68ed745a30d7" (UID: "c955564e-6a8b-4b18-b4b3-68ed745a30d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:18.435447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.435426    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c955564e-6a8b-4b18-b4b3-68ed745a30d7" (UID: "c955564e-6a8b-4b18-b4b3-68ed745a30d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:52:18.435550 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.435482    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-dshm" (OuterVolumeSpecName: "dshm") pod "c955564e-6a8b-4b18-b4b3-68ed745a30d7" (UID: "c955564e-6a8b-4b18-b4b3-68ed745a30d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:18.435550 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.435528    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c955564e-6a8b-4b18-b4b3-68ed745a30d7-kube-api-access-7g4p2" (OuterVolumeSpecName: "kube-api-access-7g4p2") pod "c955564e-6a8b-4b18-b4b3-68ed745a30d7" (UID: "c955564e-6a8b-4b18-b4b3-68ed745a30d7"). InnerVolumeSpecName "kube-api-access-7g4p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:52:18.534492 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.534454    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:18.534492 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.534487    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:18.534492 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.534495    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:18.534492 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.534505    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:18.534715 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.534513    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g4p2\" (UniqueName: \"kubernetes.io/projected/c955564e-6a8b-4b18-b4b3-68ed745a30d7-kube-api-access-7g4p2\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:18.534715 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:18.534521    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c955564e-6a8b-4b18-b4b3-68ed745a30d7-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:19.038742 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.038708    2572 generic.go:356] "Generic (PLEG): container finished" podID="c955564e-6a8b-4b18-b4b3-68ed745a30d7" containerID="edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c" exitCode=0
Jul 25 01:52:19.038924 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.038753    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"
Jul 25 01:52:19.038924 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.038762    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx" event={"ID":"c955564e-6a8b-4b18-b4b3-68ed745a30d7","Type":"ContainerDied","Data":"edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c"}
Jul 25 01:52:19.038924 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.038794    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx" event={"ID":"c955564e-6a8b-4b18-b4b3-68ed745a30d7","Type":"ContainerDied","Data":"054a9008dc6f23a0b7b2e1c163cb4220b1a527fc5ae5ff31300a7533f8195f79"}
Jul 25 01:52:19.038924 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.038813    2572 scope.go:117] "RemoveContainer" containerID="edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c"
Jul 25 01:52:19.047117 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.047063    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v" podUID="aa4f6711-c45a-4458-8e06-16bcf3694b93" containerName="main" containerID="cri-o://bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c" gracePeriod=45
Jul 25 01:52:19.048198 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.048175    2572 scope.go:117] "RemoveContainer" containerID="edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c"
Jul 25 01:52:19.048443 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:52:19.048422    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c\": container with ID starting with edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c not found: ID does not exist" containerID="edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c"
Jul 25 01:52:19.048532 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.048449    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c"} err="failed to get container status \"edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c\": rpc error: code = NotFound desc = could not find container \"edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c\": container with ID starting with edc9260959e97d5a4d3f1c39bd77b7d49bc726bc209855899c85e913e3c6a44c not found: ID does not exist"
Jul 25 01:52:19.060169 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.060145    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"]
Jul 25 01:52:19.062200 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.062176    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-df5e5d/test-model-df5e5d-kserve-79cb686f9f-w7brx"]
Jul 25 01:52:19.303297 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.303270    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:52:19.441650 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.441614    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-dshm\") pod \"aa4f6711-c45a-4458-8e06-16bcf3694b93\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") "
Jul 25 01:52:19.441844 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.441675    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-tmp-dir\") pod \"aa4f6711-c45a-4458-8e06-16bcf3694b93\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") "
Jul 25 01:52:19.441844 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.441717    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcx95\" (UniqueName: \"kubernetes.io/projected/aa4f6711-c45a-4458-8e06-16bcf3694b93-kube-api-access-jcx95\") pod \"aa4f6711-c45a-4458-8e06-16bcf3694b93\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") "
Jul 25 01:52:19.441844 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.441738    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-home\") pod \"aa4f6711-c45a-4458-8e06-16bcf3694b93\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") "
Jul 25 01:52:19.441844 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.441775    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f6711-c45a-4458-8e06-16bcf3694b93-tls-certs\") pod \"aa4f6711-c45a-4458-8e06-16bcf3694b93\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") "
Jul 25 01:52:19.441844 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.441807    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-model-cache\") pod \"aa4f6711-c45a-4458-8e06-16bcf3694b93\" (UID: \"aa4f6711-c45a-4458-8e06-16bcf3694b93\") "
Jul 25 01:52:19.442087 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.441990    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa4f6711-c45a-4458-8e06-16bcf3694b93" (UID: "aa4f6711-c45a-4458-8e06-16bcf3694b93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:19.442087 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.442021    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-home" (OuterVolumeSpecName: "home") pod "aa4f6711-c45a-4458-8e06-16bcf3694b93" (UID: "aa4f6711-c45a-4458-8e06-16bcf3694b93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:19.442174 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.442102    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa4f6711-c45a-4458-8e06-16bcf3694b93" (UID: "aa4f6711-c45a-4458-8e06-16bcf3694b93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:19.442174 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.442165    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:19.442242 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.442178    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:19.442242 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.442186    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:19.444028 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.444008    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4f6711-c45a-4458-8e06-16bcf3694b93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa4f6711-c45a-4458-8e06-16bcf3694b93" (UID: "aa4f6711-c45a-4458-8e06-16bcf3694b93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:52:19.444336 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.444307    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4f6711-c45a-4458-8e06-16bcf3694b93-kube-api-access-jcx95" (OuterVolumeSpecName: "kube-api-access-jcx95") pod "aa4f6711-c45a-4458-8e06-16bcf3694b93" (UID: "aa4f6711-c45a-4458-8e06-16bcf3694b93"). InnerVolumeSpecName "kube-api-access-jcx95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:52:19.444395 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.444316    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-dshm" (OuterVolumeSpecName: "dshm") pod "aa4f6711-c45a-4458-8e06-16bcf3694b93" (UID: "aa4f6711-c45a-4458-8e06-16bcf3694b93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:19.542839 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.542766    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa4f6711-c45a-4458-8e06-16bcf3694b93-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:19.542839 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.542793    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcx95\" (UniqueName: \"kubernetes.io/projected/aa4f6711-c45a-4458-8e06-16bcf3694b93-kube-api-access-jcx95\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:19.542839 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:19.542804    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa4f6711-c45a-4458-8e06-16bcf3694b93-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:20.046539 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:20.046501    2572 generic.go:356] "Generic (PLEG): container finished" podID="aa4f6711-c45a-4458-8e06-16bcf3694b93" containerID="bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c" exitCode=0
Jul 25 01:52:20.046716 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:20.046543    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"
Jul 25 01:52:20.046716 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:20.046589    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v" event={"ID":"aa4f6711-c45a-4458-8e06-16bcf3694b93","Type":"ContainerDied","Data":"bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c"}
Jul 25 01:52:20.046716 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:20.046624    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v" event={"ID":"aa4f6711-c45a-4458-8e06-16bcf3694b93","Type":"ContainerDied","Data":"48e841fb73c56997ef7d521e913a4b6b9ce335d7b19d272cd75e4448e0c91383"}
Jul 25 01:52:20.046716 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:20.046645    2572 scope.go:117] "RemoveContainer" containerID="bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c"
Jul 25 01:52:20.058423 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:20.058397    2572 scope.go:117] "RemoveContainer" containerID="bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c"
Jul 25 01:52:20.058714 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:52:20.058689    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c\": container with ID starting with bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c not found: ID does not exist" containerID="bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c"
Jul 25 01:52:20.058777 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:20.058725    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c"} err="failed to get container status \"bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c\": rpc error: code = NotFound desc = could not find container \"bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c\": container with ID starting with bee026b741a55f87a1ebf740db87ffe683e4a93daf318f8ce02c24cac1d6a93c not found: ID does not exist"
Jul 25 01:52:20.076979 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:20.076951    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"]
Jul 25 01:52:20.080252 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:20.080230    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-320e6c/test-model-320e6c-kserve-784dfdd5ff-j856v"]
Jul 25 01:52:20.353553 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:20.353476    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa4f6711-c45a-4458-8e06-16bcf3694b93" path="/var/lib/kubelet/pods/aa4f6711-c45a-4458-8e06-16bcf3694b93/volumes"
Jul 25 01:52:20.353884 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:20.353871    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c955564e-6a8b-4b18-b4b3-68ed745a30d7" path="/var/lib/kubelet/pods/c955564e-6a8b-4b18-b4b3-68ed745a30d7/volumes"
Jul 25 01:52:25.726883 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:25.726843    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"]
Jul 25 01:52:25.727303 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:25.727097    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy" containerID="cri-o://7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e" gracePeriod=30
Jul 25 01:52:30.981616 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:30.981586    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:52:31.026540 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026464    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-credential-socket\") pod \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") "
Jul 25 01:52:31.026540 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026496    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-data\") pod \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") "
Jul 25 01:52:31.026540 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026515    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-socket\") pod \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") "
Jul 25 01:52:31.026540 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026539    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istiod-ca-cert\") pod \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") "
Jul 25 01:52:31.026872 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026572    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-certs\") pod \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") "
Jul 25 01:52:31.026872 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026600    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-token\") pod \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") "
Jul 25 01:52:31.026872 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026637    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6f87\" (UniqueName: \"kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-kube-api-access-q6f87\") pod \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") "
Jul 25 01:52:31.026872 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026769    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8116e67f-1b04-4c53-bbce-bacfaf8333c9" (UID: "8116e67f-1b04-4c53-bbce-bacfaf8333c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:31.026872 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026786    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8116e67f-1b04-4c53-bbce-bacfaf8333c9" (UID: "8116e67f-1b04-4c53-bbce-bacfaf8333c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:31.026872 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026821    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-envoy\") pod \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") "
Jul 25 01:52:31.027546 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026877    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-podinfo\") pod \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\" (UID: \"8116e67f-1b04-4c53-bbce-bacfaf8333c9\") "
Jul 25 01:52:31.027546 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.026912    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8116e67f-1b04-4c53-bbce-bacfaf8333c9" (UID: "8116e67f-1b04-4c53-bbce-bacfaf8333c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:31.027546 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.027001    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8116e67f-1b04-4c53-bbce-bacfaf8333c9" (UID: "8116e67f-1b04-4c53-bbce-bacfaf8333c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:52:31.027546 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.027192    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istiod-ca-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:31.027546 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.027219    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:31.027546 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.027233    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-credential-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:31.027546 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.027247    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-workload-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:31.027546 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.027523    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-data" (OuterVolumeSpecName: "istio-data") pod "8116e67f-1b04-4c53-bbce-bacfaf8333c9" (UID: "8116e67f-1b04-4c53-bbce-bacfaf8333c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:31.029090 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.029063    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-kube-api-access-q6f87" (OuterVolumeSpecName: "kube-api-access-q6f87") pod "8116e67f-1b04-4c53-bbce-bacfaf8333c9" (UID: "8116e67f-1b04-4c53-bbce-bacfaf8333c9"). InnerVolumeSpecName "kube-api-access-q6f87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:52:31.029209 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.029109    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8116e67f-1b04-4c53-bbce-bacfaf8333c9" (UID: "8116e67f-1b04-4c53-bbce-bacfaf8333c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:31.029336 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.029303    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-token" (OuterVolumeSpecName: "istio-token") pod "8116e67f-1b04-4c53-bbce-bacfaf8333c9" (UID: "8116e67f-1b04-4c53-bbce-bacfaf8333c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:52:31.029470 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.029448    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8116e67f-1b04-4c53-bbce-bacfaf8333c9" (UID: "8116e67f-1b04-4c53-bbce-bacfaf8333c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 25 01:52:31.104983 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.104946    2572 generic.go:356] "Generic (PLEG): container finished" podID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerID="7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e" exitCode=0
Jul 25 01:52:31.105153 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.105002    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"
Jul 25 01:52:31.105153 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.105000    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" event={"ID":"8116e67f-1b04-4c53-bbce-bacfaf8333c9","Type":"ContainerDied","Data":"7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e"}
Jul 25 01:52:31.105153 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.105121    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk" event={"ID":"8116e67f-1b04-4c53-bbce-bacfaf8333c9","Type":"ContainerDied","Data":"c69fc937ca45f8ab70428718566f55bd8fce1ece869b727695ddb2c14db68e6f"}
Jul 25 01:52:31.105153 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.105151    2572 scope.go:117] "RemoveContainer" containerID="7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e"
Jul 25 01:52:31.117393 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.117375    2572 scope.go:117] "RemoveContainer" containerID="e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4"
Jul 25 01:52:31.125282 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.125265    2572 scope.go:117] "RemoveContainer" containerID="7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e"
Jul 25 01:52:31.125539 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:52:31.125519    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e\": container with ID starting with 7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e not found: ID does not exist" containerID="7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e"
Jul 25 01:52:31.125610 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.125546    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e"} err="failed to get container status \"7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e\": rpc error: code = NotFound desc = could not find container \"7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e\": container with ID starting with 7160e4f3ff87844ffb02f0fd48132d4c2f7527f163a401e1b6947a9787d8692e not found: ID does not exist"
Jul 25 01:52:31.125610 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.125562    2572 scope.go:117] "RemoveContainer" containerID="e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4"
Jul 25 01:52:31.125795 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:52:31.125778    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4\": container with ID starting with e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4 not found: ID does not exist" containerID="e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4"
Jul 25 01:52:31.125856 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.125799    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4"} err="failed to get container status \"e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4\": rpc error: code = NotFound desc = could not find container \"e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4\": container with ID starting with e9b0534c3c5dc06ad3586aa5aeb1204a39a8e2c1a9c25f982a93bd0c5fe225c4 not found: ID does not exist"
Jul 25 01:52:31.127452 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.127436    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"]
Jul 25 01:52:31.127748 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.127728    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-data\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:31.127822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.127755    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-token\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:31.127822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.127769    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6f87\" (UniqueName: \"kubernetes.io/projected/8116e67f-1b04-4c53-bbce-bacfaf8333c9-kube-api-access-q6f87\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:31.127822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.127784    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-envoy\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:31.127822 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.127798    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8116e67f-1b04-4c53-bbce-bacfaf8333c9-istio-podinfo\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:31.135583 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:31.135565    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-df5e5d-openshift-default-569f88687b-6rwqk"]
Jul 25 01:52:32.356259 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:32.356181    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" path="/var/lib/kubelet/pods/8116e67f-1b04-4c53-bbce-bacfaf8333c9/volumes"
Jul 25 01:52:37.068714 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.068680    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"]
Jul 25 01:52:37.069270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.068906    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk" podUID="f25828b2-87fd-4194-a872-e179cc80f599" containerName="maas-api" containerID="cri-o://37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5" gracePeriod=30
Jul 25 01:52:37.310637 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.310610    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:52:37.374669 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.374585    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb69d\" (UniqueName: \"kubernetes.io/projected/f25828b2-87fd-4194-a872-e179cc80f599-kube-api-access-sb69d\") pod \"f25828b2-87fd-4194-a872-e179cc80f599\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") "
Jul 25 01:52:37.374669 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.374639    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f25828b2-87fd-4194-a872-e179cc80f599-maas-api-tls\") pod \"f25828b2-87fd-4194-a872-e179cc80f599\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") "
Jul 25 01:52:37.374669 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.374661    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25828b2-87fd-4194-a872-e179cc80f599-openshift-service-ca\") pod \"f25828b2-87fd-4194-a872-e179cc80f599\" (UID: \"f25828b2-87fd-4194-a872-e179cc80f599\") "
Jul 25 01:52:37.375073 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.375044    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25828b2-87fd-4194-a872-e179cc80f599-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f25828b2-87fd-4194-a872-e179cc80f599" (UID: "f25828b2-87fd-4194-a872-e179cc80f599"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:52:37.376759 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.376733    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25828b2-87fd-4194-a872-e179cc80f599-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f25828b2-87fd-4194-a872-e179cc80f599" (UID: "f25828b2-87fd-4194-a872-e179cc80f599"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:52:37.376858 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.376770    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25828b2-87fd-4194-a872-e179cc80f599-kube-api-access-sb69d" (OuterVolumeSpecName: "kube-api-access-sb69d") pod "f25828b2-87fd-4194-a872-e179cc80f599" (UID: "f25828b2-87fd-4194-a872-e179cc80f599"). InnerVolumeSpecName "kube-api-access-sb69d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:52:37.475392 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.475346    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f25828b2-87fd-4194-a872-e179cc80f599-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:37.475392 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.475388    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f25828b2-87fd-4194-a872-e179cc80f599-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:37.475392 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:37.475399    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb69d\" (UniqueName: \"kubernetes.io/projected/f25828b2-87fd-4194-a872-e179cc80f599-kube-api-access-sb69d\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:38.143123 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:38.143086    2572 generic.go:356] "Generic (PLEG): container finished" podID="f25828b2-87fd-4194-a872-e179cc80f599" containerID="37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5" exitCode=0
Jul 25 01:52:38.143580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:38.143132    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk" event={"ID":"f25828b2-87fd-4194-a872-e179cc80f599","Type":"ContainerDied","Data":"37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5"}
Jul 25 01:52:38.143580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:38.143151    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"
Jul 25 01:52:38.143580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:38.143217    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk" event={"ID":"f25828b2-87fd-4194-a872-e179cc80f599","Type":"ContainerDied","Data":"b00782c9eabaed7fab7abeca0443f06f8dfd928015d2d1a3cfe7df1285801069"}
Jul 25 01:52:38.143580 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:38.143250    2572 scope.go:117] "RemoveContainer" containerID="37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5"
Jul 25 01:52:38.152441 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:38.152422    2572 scope.go:117] "RemoveContainer" containerID="37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5"
Jul 25 01:52:38.152664 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:52:38.152645    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5\": container with ID starting with 37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5 not found: ID does not exist" containerID="37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5"
Jul 25 01:52:38.152713 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:38.152673    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5"} err="failed to get container status \"37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5\": rpc error: code = NotFound desc = could not find container \"37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5\": container with ID starting with 37fe7a96835cc469652376e69a78683f0b57be266353dc3014364fadd78b1ea5 not found: ID does not exist"
Jul 25 01:52:38.164314 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:38.164279    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"]
Jul 25 01:52:38.169613 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:38.169588    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-320e6c-7597c498c7-26ldk"]
Jul 25 01:52:38.352608 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:38.352574    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f25828b2-87fd-4194-a872-e179cc80f599" path="/var/lib/kubelet/pods/f25828b2-87fd-4194-a872-e179cc80f599/volumes"
Jul 25 01:52:47.650240 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:47.650202    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"]
Jul 25 01:52:47.650636 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:47.650473    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerName="istio-proxy" containerID="cri-o://15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8" gracePeriod=30
Jul 25 01:52:48.549130 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549092    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"]
Jul 25 01:52:48.549442 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549429    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c955564e-6a8b-4b18-b4b3-68ed745a30d7" containerName="main"
Jul 25 01:52:48.549491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549443    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c955564e-6a8b-4b18-b4b3-68ed745a30d7" containerName="main"
Jul 25 01:52:48.549491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549456    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy"
Jul 25 01:52:48.549491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549461    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy"
Jul 25 01:52:48.549491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549468    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy"
Jul 25 01:52:48.549491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549474    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy"
Jul 25 01:52:48.549491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549479    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3054967b-e564-4455-8c72-7f0bea819952" containerName="maas-api"
Jul 25 01:52:48.549491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549485    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3054967b-e564-4455-8c72-7f0bea819952" containerName="maas-api"
Jul 25 01:52:48.549705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549497    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f25828b2-87fd-4194-a872-e179cc80f599" containerName="maas-api"
Jul 25 01:52:48.549705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549502    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f25828b2-87fd-4194-a872-e179cc80f599" containerName="maas-api"
Jul 25 01:52:48.549705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549509    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa4f6711-c45a-4458-8e06-16bcf3694b93" containerName="main"
Jul 25 01:52:48.549705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549514    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa4f6711-c45a-4458-8e06-16bcf3694b93" containerName="main"
Jul 25 01:52:48.549705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549564    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c955564e-6a8b-4b18-b4b3-68ed745a30d7" containerName="main"
Jul 25 01:52:48.549705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549572    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy"
Jul 25 01:52:48.549705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549577    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa4f6711-c45a-4458-8e06-16bcf3694b93" containerName="main"
Jul 25 01:52:48.549705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549583    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3054967b-e564-4455-8c72-7f0bea819952" containerName="maas-api"
Jul 25 01:52:48.549705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.549589    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f25828b2-87fd-4194-a872-e179cc80f599" containerName="maas-api"
Jul 25 01:52:48.553727 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.553707    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.556020 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.556001    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-88f354-openshift-default-dockercfg-7z9qj\""
Jul 25 01:52:48.563475 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.563453    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"]
Jul 25 01:52:48.667806 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.667773    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-credential-socket\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.668223 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.667813    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-data\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.668223 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.667835    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-token\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.668223 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.667936    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-socket\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.668223 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.667965    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-envoy\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.668223 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.667985    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-podinfo\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.668223 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.668012    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istiod-ca-cert\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.668223 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.668056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7fj\" (UniqueName: \"kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-kube-api-access-5z7fj\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.668223 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.668083    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-certs\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769028 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.768990    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-socket\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769213 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769069    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-envoy\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769213 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769161    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-podinfo\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769213 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769199    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istiod-ca-cert\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769407 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769222    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7fj\" (UniqueName: \"kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-kube-api-access-5z7fj\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769407 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769254    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-certs\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769407 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769334    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-credential-socket\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769572 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769438    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-data\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769572 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769476    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-token\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769705 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769611    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-certs\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769760 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769707    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-credential-socket\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769823 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769806    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-socket\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769901 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769875    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-data\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.769984 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.769964    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istiod-ca-cert\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.771360 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.771317    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-envoy\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.771651 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.771635    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-podinfo\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.776723 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.776697    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7fj\" (UniqueName: \"kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-kube-api-access-5z7fj\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.776913 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.776891    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-token\") pod \"e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") " pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.865397 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.865311    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:48.995045 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:48.995014    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"]
Jul 25 01:52:48.996429 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:52:48.996402    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bf23a5_ce06_4842_a623_aa280b1ed9ae.slice/crio-b0ae7d568188ce0a171648670673c1959d27bbd39878ed0a096b63fc8df43231 WatchSource:0}: Error finding container b0ae7d568188ce0a171648670673c1959d27bbd39878ed0a096b63fc8df43231: Status 404 returned error can't find the container with id b0ae7d568188ce0a171648670673c1959d27bbd39878ed0a096b63fc8df43231
Jul 25 01:52:49.001589 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:49.001556    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:52:49.001679 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:49.001620    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:52:49.001679 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:49.001648    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:52:49.199421 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:49.199381    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" event={"ID":"57bf23a5-ce06-4842-a623-aa280b1ed9ae","Type":"ContainerStarted","Data":"4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642"}
Jul 25 01:52:49.199625 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:49.199438    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" event={"ID":"57bf23a5-ce06-4842-a623-aa280b1ed9ae","Type":"ContainerStarted","Data":"b0ae7d568188ce0a171648670673c1959d27bbd39878ed0a096b63fc8df43231"}
Jul 25 01:52:49.224467 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:49.219041    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" podStartSLOduration=1.219018942 podStartE2EDuration="1.219018942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:52:49.215925649 +0000 UTC m=+2453.325734944" watchObservedRunningTime="2026-07-25 01:52:49.219018942 +0000 UTC m=+2453.328828226"
Jul 25 01:52:49.865772 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:49.865741    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:50.871029 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:50.870996    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:51.211215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.211184    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"]
Jul 25 01:52:51.211780 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.211763    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8116e67f-1b04-4c53-bbce-bacfaf8333c9" containerName="istio-proxy"
Jul 25 01:52:51.215057 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.215035    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:51.215175 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.215105    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:52:51.215270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.215253    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:51.217231 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.217205    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-88f354-serving-cert\""
Jul 25 01:52:51.217380 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.217346    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhv77\""
Jul 25 01:52:51.225451 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.225432    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"]
Jul 25 01:52:51.292988 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.292951    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff45d19-93a0-42ab-81d4-db5252535f36-maas-api-tls\") pod \"maas-api-e2e-api-a-88f354-68dcf89448-7t5tz\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:51.293158 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.293032    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tm26\" (UniqueName: \"kubernetes.io/projected/aff45d19-93a0-42ab-81d4-db5252535f36-kube-api-access-7tm26\") pod \"maas-api-e2e-api-a-88f354-68dcf89448-7t5tz\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:51.293158 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.293073    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff45d19-93a0-42ab-81d4-db5252535f36-openshift-service-ca\") pod \"maas-api-e2e-api-a-88f354-68dcf89448-7t5tz\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:51.393920 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.393875    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tm26\" (UniqueName: \"kubernetes.io/projected/aff45d19-93a0-42ab-81d4-db5252535f36-kube-api-access-7tm26\") pod \"maas-api-e2e-api-a-88f354-68dcf89448-7t5tz\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:51.394138 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.393948    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff45d19-93a0-42ab-81d4-db5252535f36-openshift-service-ca\") pod \"maas-api-e2e-api-a-88f354-68dcf89448-7t5tz\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:51.394215 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.394137    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff45d19-93a0-42ab-81d4-db5252535f36-maas-api-tls\") pod \"maas-api-e2e-api-a-88f354-68dcf89448-7t5tz\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:51.394698 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.394674    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff45d19-93a0-42ab-81d4-db5252535f36-openshift-service-ca\") pod \"maas-api-e2e-api-a-88f354-68dcf89448-7t5tz\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:51.396510 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.396489    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff45d19-93a0-42ab-81d4-db5252535f36-maas-api-tls\") pod \"maas-api-e2e-api-a-88f354-68dcf89448-7t5tz\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:51.407691 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.407668    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tm26\" (UniqueName: \"kubernetes.io/projected/aff45d19-93a0-42ab-81d4-db5252535f36-kube-api-access-7tm26\") pod \"maas-api-e2e-api-a-88f354-68dcf89448-7t5tz\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:51.528319 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.528252    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:51.659788 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.659759    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"]
Jul 25 01:52:51.661169 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:52:51.661143    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff45d19_93a0_42ab_81d4_db5252535f36.slice/crio-e509ab3063cab5c63b7e8e73878a14050757291b08468943db80b51e9c82565a WatchSource:0}: Error finding container e509ab3063cab5c63b7e8e73878a14050757291b08468943db80b51e9c82565a: Status 404 returned error can't find the container with id e509ab3063cab5c63b7e8e73878a14050757291b08468943db80b51e9c82565a
Jul 25 01:52:51.662655 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:51.662635    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 25 01:52:52.218270 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:52.218231    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz" event={"ID":"aff45d19-93a0-42ab-81d4-db5252535f36","Type":"ContainerStarted","Data":"e509ab3063cab5c63b7e8e73878a14050757291b08468943db80b51e9c82565a"}
Jul 25 01:52:52.904220 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:52.904192    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:52:53.009000 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.008898    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istiod-ca-cert\") pod \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") "
Jul 25 01:52:53.009000 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.008959    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-envoy\") pod \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") "
Jul 25 01:52:53.009000 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.008978    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-token\") pod \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") "
Jul 25 01:52:53.009296 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009061    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-data\") pod \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") "
Jul 25 01:52:53.009296 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009112    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-credential-socket\") pod \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") "
Jul 25 01:52:53.009296 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009161    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-socket\") pod \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") "
Jul 25 01:52:53.009296 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009187    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-certs\") pod \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") "
Jul 25 01:52:53.009296 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009223    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6lcw\" (UniqueName: \"kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-kube-api-access-s6lcw\") pod \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") "
Jul 25 01:52:53.009296 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009287    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-podinfo\") pod \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\" (UID: \"e72f9283-1fe4-43b7-9614-fc1a0f4e7964\") "
Jul 25 01:52:53.009643 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009304    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e72f9283-1fe4-43b7-9614-fc1a0f4e7964" (UID: "e72f9283-1fe4-43b7-9614-fc1a0f4e7964"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:53.009643 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009375    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e72f9283-1fe4-43b7-9614-fc1a0f4e7964" (UID: "e72f9283-1fe4-43b7-9614-fc1a0f4e7964"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:52:53.009643 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009469    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e72f9283-1fe4-43b7-9614-fc1a0f4e7964" (UID: "e72f9283-1fe4-43b7-9614-fc1a0f4e7964"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:53.009643 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009577    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e72f9283-1fe4-43b7-9614-fc1a0f4e7964" (UID: "e72f9283-1fe4-43b7-9614-fc1a0f4e7964"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:53.009643 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009629    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istiod-ca-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:53.009907 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009647    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-credential-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:53.009907 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009662    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:53.009907 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.009816    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-data" (OuterVolumeSpecName: "istio-data") pod "e72f9283-1fe4-43b7-9614-fc1a0f4e7964" (UID: "e72f9283-1fe4-43b7-9614-fc1a0f4e7964"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:53.011660 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.011632    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-token" (OuterVolumeSpecName: "istio-token") pod "e72f9283-1fe4-43b7-9614-fc1a0f4e7964" (UID: "e72f9283-1fe4-43b7-9614-fc1a0f4e7964"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:52:53.011795 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.011670    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e72f9283-1fe4-43b7-9614-fc1a0f4e7964" (UID: "e72f9283-1fe4-43b7-9614-fc1a0f4e7964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:52:53.011916 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.011890    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e72f9283-1fe4-43b7-9614-fc1a0f4e7964" (UID: "e72f9283-1fe4-43b7-9614-fc1a0f4e7964"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 25 01:52:53.012025 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.011969    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-kube-api-access-s6lcw" (OuterVolumeSpecName: "kube-api-access-s6lcw") pod "e72f9283-1fe4-43b7-9614-fc1a0f4e7964" (UID: "e72f9283-1fe4-43b7-9614-fc1a0f4e7964"). InnerVolumeSpecName "kube-api-access-s6lcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:52:53.110437 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.110402    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-envoy\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:53.110437 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.110431    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-token\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:53.110437 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.110442    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-data\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:53.110670 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.110450    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-workload-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:53.110670 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.110459    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6lcw\" (UniqueName: \"kubernetes.io/projected/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-kube-api-access-s6lcw\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:53.110670 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.110469    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e72f9283-1fe4-43b7-9614-fc1a0f4e7964-istio-podinfo\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:52:53.225146 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.225107    2572 generic.go:356] "Generic (PLEG): container finished" podID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerID="15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8" exitCode=0
Jul 25 01:52:53.225616 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.225156    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"
Jul 25 01:52:53.225616 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.225190    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" event={"ID":"e72f9283-1fe4-43b7-9614-fc1a0f4e7964","Type":"ContainerDied","Data":"15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8"}
Jul 25 01:52:53.225616 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.225240    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t" event={"ID":"e72f9283-1fe4-43b7-9614-fc1a0f4e7964","Type":"ContainerDied","Data":"9c077fa099a00a40a23e37c7f39405afdf0d74fd5c74e85ab0220493718572fa"}
Jul 25 01:52:53.225616 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.225265    2572 scope.go:117] "RemoveContainer" containerID="15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8"
Jul 25 01:52:53.226893 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.226867    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz" event={"ID":"aff45d19-93a0-42ab-81d4-db5252535f36","Type":"ContainerStarted","Data":"57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a"}
Jul 25 01:52:53.234570 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.234554    2572 scope.go:117] "RemoveContainer" containerID="874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307"
Jul 25 01:52:53.242457 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.242443    2572 scope.go:117] "RemoveContainer" containerID="15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8"
Jul 25 01:52:53.242705 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:52:53.242686    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8\": container with ID starting with 15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8 not found: ID does not exist" containerID="15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8"
Jul 25 01:52:53.242743 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.242716    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8"} err="failed to get container status \"15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8\": rpc error: code = NotFound desc = could not find container \"15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8\": container with ID starting with 15596d37a5dc9481332fd2c0a1b5fec1e4c8d2ea2cf73d709283e66b09b6d8b8 not found: ID does not exist"
Jul 25 01:52:53.242743 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.242731    2572 scope.go:117] "RemoveContainer" containerID="874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307"
Jul 25 01:52:53.242960 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:52:53.242944    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307\": container with ID starting with 874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307 not found: ID does not exist" containerID="874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307"
Jul 25 01:52:53.243008 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.242963    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307"} err="failed to get container status \"874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307\": rpc error: code = NotFound desc = could not find container \"874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307\": container with ID starting with 874c4c0f6c72a0ad6bcf41a195008cdf7a416c3e7d43749af51ea07b2244a307 not found: ID does not exist"
Jul 25 01:52:53.245846 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.245807    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz" podStartSLOduration=1.685517558 podStartE2EDuration="2.245794467s" totalImagesPullingTime="560.276909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:52:53.243893167 +0000 UTC m=+2457.353702453" watchObservedRunningTime="2026-07-25 01:52:53.245794467 +0000 UTC m=+2457.355603759"
Jul 25 01:52:53.256678 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.256656    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"]
Jul 25 01:52:53.260205 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:53.260156    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-320e6c-openshift-default-6769647446-khs4t"]
Jul 25 01:52:54.232499 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:54.232459    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:52:54.352547 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:54.352512    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" path="/var/lib/kubelet/pods/e72f9283-1fe4-43b7-9614-fc1a0f4e7964/volumes"
Jul 25 01:52:59.535401 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.535370    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"]
Jul 25 01:52:59.535860 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.535850    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerName="istio-proxy"
Jul 25 01:52:59.535922 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.535865    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerName="istio-proxy"
Jul 25 01:52:59.535995 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.535970    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerName="istio-proxy"
Jul 25 01:52:59.541015 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.540994    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:52:59.543375 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.543348    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-393e3a-serving-cert\""
Jul 25 01:52:59.548870 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.548846    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"]
Jul 25 01:52:59.660738 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.660705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkvjl\" (UniqueName: \"kubernetes.io/projected/77f82d27-27a8-4df6-acea-3bec3116f608-kube-api-access-xkvjl\") pod \"maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:52:59.660940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.660766    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f82d27-27a8-4df6-acea-3bec3116f608-maas-api-tls\") pod \"maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:52:59.660940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.660819    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f82d27-27a8-4df6-acea-3bec3116f608-openshift-service-ca\") pod \"maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:52:59.761278 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.761248    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvjl\" (UniqueName: \"kubernetes.io/projected/77f82d27-27a8-4df6-acea-3bec3116f608-kube-api-access-xkvjl\") pod \"maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:52:59.761450 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.761306    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f82d27-27a8-4df6-acea-3bec3116f608-maas-api-tls\") pod \"maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:52:59.761450 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.761362    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f82d27-27a8-4df6-acea-3bec3116f608-openshift-service-ca\") pod \"maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:52:59.762065 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.762046    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f82d27-27a8-4df6-acea-3bec3116f608-openshift-service-ca\") pod \"maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:52:59.763889 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.763863    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f82d27-27a8-4df6-acea-3bec3116f608-maas-api-tls\") pod \"maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:52:59.770100 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.770069    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvjl\" (UniqueName: \"kubernetes.io/projected/77f82d27-27a8-4df6-acea-3bec3116f608-kube-api-access-xkvjl\") pod \"maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:52:59.853481 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.853389    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:52:59.988993 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:52:59.988961    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"]
Jul 25 01:52:59.990723 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:52:59.990687    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f82d27_27a8_4df6_acea_3bec3116f608.slice/crio-8438b7c78be7d2036c6349e6037dedd6190cb55aa0f8c486230bd97dcf29dd5f WatchSource:0}: Error finding container 8438b7c78be7d2036c6349e6037dedd6190cb55aa0f8c486230bd97dcf29dd5f: Status 404 returned error can't find the container with id 8438b7c78be7d2036c6349e6037dedd6190cb55aa0f8c486230bd97dcf29dd5f
Jul 25 01:53:00.244620 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:00.244576    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:53:00.263795 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:00.263765    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v" event={"ID":"77f82d27-27a8-4df6-acea-3bec3116f608","Type":"ContainerStarted","Data":"8438b7c78be7d2036c6349e6037dedd6190cb55aa0f8c486230bd97dcf29dd5f"}
Jul 25 01:53:01.272394 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:01.272352    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v" event={"ID":"77f82d27-27a8-4df6-acea-3bec3116f608","Type":"ContainerStarted","Data":"032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f"}
Jul 25 01:53:01.272812 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:01.272502    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:53:01.288816 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:01.288762    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v" podStartSLOduration=1.904719616 podStartE2EDuration="2.288744725s" totalImagesPullingTime="384.025109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:53:01.286056624 +0000 UTC m=+2465.395865909" watchObservedRunningTime="2026-07-25 01:53:01.288744725 +0000 UTC m=+2465.398554054"
Jul 25 01:53:07.255471 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:07.255428    2572 patch_prober.go:28] interesting pod/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 25 01:53:07.255873 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:07.255516    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 25 01:53:07.285312 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:07.285283    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:53:08.312094 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:08.312059    2572 generic.go:356] "Generic (PLEG): container finished" podID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerID="4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642" exitCode=0
Jul 25 01:53:08.312436 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:08.312130    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" event={"ID":"57bf23a5-ce06-4842-a623-aa280b1ed9ae","Type":"ContainerDied","Data":"4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642"}
Jul 25 01:53:08.312535 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:08.312520    2572 scope.go:117] "RemoveContainer" containerID="4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642"
Jul 25 01:53:08.313529 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:08.313497    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:53:08.313593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:08.313580    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:53:08.313633 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:08.313622    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:53:09.319848 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:09.319811    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" event={"ID":"57bf23a5-ce06-4842-a623-aa280b1ed9ae","Type":"ContainerStarted","Data":"fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb"}
Jul 25 01:53:09.866412 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:09.866373    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:53:09.866595 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:09.866569    2572 patch_prober.go:28] interesting pod/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 25 01:53:09.866640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:09.866615    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 25 01:53:10.866166 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:10.866124    2572 patch_prober.go:28] interesting pod/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 25 01:53:10.866683 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:10.866183    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 25 01:53:11.865775 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:11.865725    2572 patch_prober.go:28] interesting pod/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 25 01:53:11.865950 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:11.865794    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 25 01:53:12.199836 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.199795    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"]
Jul 25 01:53:12.200368 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.200349    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerName="istio-proxy"
Jul 25 01:53:12.200459 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.200371    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerName="istio-proxy"
Jul 25 01:53:12.200516 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.200493    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e72f9283-1fe4-43b7-9614-fc1a0f4e7964" containerName="istio-proxy"
Jul 25 01:53:12.202491 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.202469    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"
Jul 25 01:53:12.204696 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.204671    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 01:53:12.210529 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.210507    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"]
Jul 25 01:53:12.273554 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.273521    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd91d785-805d-4029-9e60-295216393ef8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88f354-qg5wg\" (UID: \"fd91d785-805d-4029-9e60-295216393ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"
Jul 25 01:53:12.374970 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.374929    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd91d785-805d-4029-9e60-295216393ef8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88f354-qg5wg\" (UID: \"fd91d785-805d-4029-9e60-295216393ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"
Jul 25 01:53:12.375593 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.375569    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd91d785-805d-4029-9e60-295216393ef8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88f354-qg5wg\" (UID: \"fd91d785-805d-4029-9e60-295216393ef8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"
Jul 25 01:53:12.514394 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.514283    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"
Jul 25 01:53:12.694795 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.694769    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"]
Jul 25 01:53:12.696612 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:53:12.696571    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd91d785_805d_4029_9e60_295216393ef8.slice/crio-d60e12ab327f25fcf8b3d0e416f9f8ae92f1c7e8e58b8caa42221463cd0cce6a WatchSource:0}: Error finding container d60e12ab327f25fcf8b3d0e416f9f8ae92f1c7e8e58b8caa42221463cd0cce6a: Status 404 returned error can't find the container with id d60e12ab327f25fcf8b3d0e416f9f8ae92f1c7e8e58b8caa42221463cd0cce6a
Jul 25 01:53:12.868280 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.868236    2572 patch_prober.go:28] interesting pod/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 25 01:53:12.869230 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:12.868707    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 25 01:53:13.341605 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:13.341559    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg" event={"ID":"fd91d785-805d-4029-9e60-295216393ef8","Type":"ContainerStarted","Data":"7f42f771df13ca6a39f4c672f6bba02901e2b5e0d5c1470a472d816875261e1a"}
Jul 25 01:53:13.341992 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:13.341674    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg" event={"ID":"fd91d785-805d-4029-9e60-295216393ef8","Type":"ContainerStarted","Data":"d60e12ab327f25fcf8b3d0e416f9f8ae92f1c7e8e58b8caa42221463cd0cce6a"}
Jul 25 01:53:13.356740 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:13.356676    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg" podStartSLOduration=1.356654979 podStartE2EDuration="1.356654979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:53:13.353954991 +0000 UTC m=+2477.463764286" watchObservedRunningTime="2026-07-25 01:53:13.356654979 +0000 UTC m=+2477.466464264"
Jul 25 01:53:13.870091 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:13.870056    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:53:13.870592 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:13.870573    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:53:13.871265 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:13.871244    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:53:14.349495 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:14.349460    2572 generic.go:356] "Generic (PLEG): container finished" podID="fd91d785-805d-4029-9e60-295216393ef8" containerID="7f42f771df13ca6a39f4c672f6bba02901e2b5e0d5c1470a472d816875261e1a" exitCode=0
Jul 25 01:53:14.352670 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:14.352640    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg" event={"ID":"fd91d785-805d-4029-9e60-295216393ef8","Type":"ContainerDied","Data":"7f42f771df13ca6a39f4c672f6bba02901e2b5e0d5c1470a472d816875261e1a"}
Jul 25 01:53:15.481619 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:15.481597    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"
Jul 25 01:53:15.603541 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:15.603505    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd91d785-805d-4029-9e60-295216393ef8-maas-api-service-ca\") pod \"fd91d785-805d-4029-9e60-295216393ef8\" (UID: \"fd91d785-805d-4029-9e60-295216393ef8\") "
Jul 25 01:53:15.603888 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:15.603863    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd91d785-805d-4029-9e60-295216393ef8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd91d785-805d-4029-9e60-295216393ef8" (UID: "fd91d785-805d-4029-9e60-295216393ef8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:53:15.705077 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:15.705045    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd91d785-805d-4029-9e60-295216393ef8-maas-api-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:16.361383 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:16.361339    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg" event={"ID":"fd91d785-805d-4029-9e60-295216393ef8","Type":"ContainerDied","Data":"d60e12ab327f25fcf8b3d0e416f9f8ae92f1c7e8e58b8caa42221463cd0cce6a"}
Jul 25 01:53:16.361513 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:16.361394    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"
Jul 25 01:53:16.361513 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:16.361399    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d60e12ab327f25fcf8b3d0e416f9f8ae92f1c7e8e58b8caa42221463cd0cce6a"
Jul 25 01:53:27.821920 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:27.821880    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"]
Jul 25 01:53:27.822473 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:27.822135    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz" podUID="aff45d19-93a0-42ab-81d4-db5252535f36" containerName="maas-api" containerID="cri-o://57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a" gracePeriod=30
Jul 25 01:53:28.066727 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.066698    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:53:28.211118 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.211085    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff45d19-93a0-42ab-81d4-db5252535f36-maas-api-tls\") pod \"aff45d19-93a0-42ab-81d4-db5252535f36\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") "
Jul 25 01:53:28.211285 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.211124    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff45d19-93a0-42ab-81d4-db5252535f36-openshift-service-ca\") pod \"aff45d19-93a0-42ab-81d4-db5252535f36\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") "
Jul 25 01:53:28.211285 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.211165    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tm26\" (UniqueName: \"kubernetes.io/projected/aff45d19-93a0-42ab-81d4-db5252535f36-kube-api-access-7tm26\") pod \"aff45d19-93a0-42ab-81d4-db5252535f36\" (UID: \"aff45d19-93a0-42ab-81d4-db5252535f36\") "
Jul 25 01:53:28.211566 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.211526    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff45d19-93a0-42ab-81d4-db5252535f36-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aff45d19-93a0-42ab-81d4-db5252535f36" (UID: "aff45d19-93a0-42ab-81d4-db5252535f36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:53:28.213117 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.213094    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff45d19-93a0-42ab-81d4-db5252535f36-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aff45d19-93a0-42ab-81d4-db5252535f36" (UID: "aff45d19-93a0-42ab-81d4-db5252535f36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:53:28.213221 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.213186    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff45d19-93a0-42ab-81d4-db5252535f36-kube-api-access-7tm26" (OuterVolumeSpecName: "kube-api-access-7tm26") pod "aff45d19-93a0-42ab-81d4-db5252535f36" (UID: "aff45d19-93a0-42ab-81d4-db5252535f36"). InnerVolumeSpecName "kube-api-access-7tm26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:53:28.312444 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.312410    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aff45d19-93a0-42ab-81d4-db5252535f36-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:28.312444 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.312438    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff45d19-93a0-42ab-81d4-db5252535f36-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:28.312444 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.312448    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tm26\" (UniqueName: \"kubernetes.io/projected/aff45d19-93a0-42ab-81d4-db5252535f36-kube-api-access-7tm26\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:28.419786 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.419750    2572 generic.go:356] "Generic (PLEG): container finished" podID="aff45d19-93a0-42ab-81d4-db5252535f36" containerID="57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a" exitCode=0
Jul 25 01:53:28.419958 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.419791    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"
Jul 25 01:53:28.419958 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.419812    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz" event={"ID":"aff45d19-93a0-42ab-81d4-db5252535f36","Type":"ContainerDied","Data":"57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a"}
Jul 25 01:53:28.419958 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.419862    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz" event={"ID":"aff45d19-93a0-42ab-81d4-db5252535f36","Type":"ContainerDied","Data":"e509ab3063cab5c63b7e8e73878a14050757291b08468943db80b51e9c82565a"}
Jul 25 01:53:28.419958 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.419894    2572 scope.go:117] "RemoveContainer" containerID="57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a"
Jul 25 01:53:28.428450 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.428436    2572 scope.go:117] "RemoveContainer" containerID="57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a"
Jul 25 01:53:28.428696 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:53:28.428676    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a\": container with ID starting with 57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a not found: ID does not exist" containerID="57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a"
Jul 25 01:53:28.428756 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.428705    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a"} err="failed to get container status \"57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a\": rpc error: code = NotFound desc = could not find container \"57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a\": container with ID starting with 57ff30580e3f02c1d748c52e0258c3cd356e53804d2fcc4cbe9cc35f1ec4982a not found: ID does not exist"
Jul 25 01:53:28.436591 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.436567    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"]
Jul 25 01:53:28.440547 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:28.440525    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-88f354-68dcf89448-7t5tz"]
Jul 25 01:53:30.353063 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:30.353023    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aff45d19-93a0-42ab-81d4-db5252535f36" path="/var/lib/kubelet/pods/aff45d19-93a0-42ab-81d4-db5252535f36/volumes"
Jul 25 01:53:32.280145 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:32.280027    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"]
Jul 25 01:53:32.281558 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:32.281528    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88f354-qg5wg"]
Jul 25 01:53:32.352833 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:32.352801    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd91d785-805d-4029-9e60-295216393ef8" path="/var/lib/kubelet/pods/fd91d785-805d-4029-9e60-295216393ef8/volumes"
Jul 25 01:53:38.776817 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:38.776784    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"]
Jul 25 01:53:38.777310 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:38.776974    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy" containerID="cri-o://fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb" gracePeriod=30
Jul 25 01:53:39.167639 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.167597    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"]
Jul 25 01:53:39.168742 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.168715    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff45d19-93a0-42ab-81d4-db5252535f36" containerName="maas-api"
Jul 25 01:53:39.168896 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.168884    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff45d19-93a0-42ab-81d4-db5252535f36" containerName="maas-api"
Jul 25 01:53:39.168983 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.168973    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd91d785-805d-4029-9e60-295216393ef8" containerName="revoke-keys"
Jul 25 01:53:39.169058 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.169049    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd91d785-805d-4029-9e60-295216393ef8" containerName="revoke-keys"
Jul 25 01:53:39.169340 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.169310    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd91d785-805d-4029-9e60-295216393ef8" containerName="revoke-keys"
Jul 25 01:53:39.169459 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.169448    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff45d19-93a0-42ab-81d4-db5252535f36" containerName="maas-api"
Jul 25 01:53:39.173070 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.173039    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"
Jul 25 01:53:39.175446 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.175424    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 01:53:39.175854 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.175826    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"]
Jul 25 01:53:39.193114 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.193088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059848e5-d957-47e3-b3f6-1343330bb458-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-393e3a-49vmd\" (UID: \"059848e5-d957-47e3-b3f6-1343330bb458\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"
Jul 25 01:53:39.293881 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.293855    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059848e5-d957-47e3-b3f6-1343330bb458-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-393e3a-49vmd\" (UID: \"059848e5-d957-47e3-b3f6-1343330bb458\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"
Jul 25 01:53:39.294433 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.294416    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059848e5-d957-47e3-b3f6-1343330bb458-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-393e3a-49vmd\" (UID: \"059848e5-d957-47e3-b3f6-1343330bb458\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"
Jul 25 01:53:39.487057 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.486958    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"
Jul 25 01:53:39.619886 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:39.619859    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"]
Jul 25 01:53:39.621446 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:53:39.621413    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059848e5_d957_47e3_b3f6_1343330bb458.slice/crio-4dcc0adb2668d8fae672b2689d99fb94a1f99d2d9a8d7d381babe61c4c0b045d WatchSource:0}: Error finding container 4dcc0adb2668d8fae672b2689d99fb94a1f99d2d9a8d7d381babe61c4c0b045d: Status 404 returned error can't find the container with id 4dcc0adb2668d8fae672b2689d99fb94a1f99d2d9a8d7d381babe61c4c0b045d
Jul 25 01:53:40.480342 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:40.480297    2572 generic.go:356] "Generic (PLEG): container finished" podID="059848e5-d957-47e3-b3f6-1343330bb458" containerID="3d7f732f5ce04db993d871e389c2f7bf6a43c3bfd99c610858036e8c792eb4d9" exitCode=0
Jul 25 01:53:40.480762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:40.480363    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd" event={"ID":"059848e5-d957-47e3-b3f6-1343330bb458","Type":"ContainerDied","Data":"3d7f732f5ce04db993d871e389c2f7bf6a43c3bfd99c610858036e8c792eb4d9"}
Jul 25 01:53:40.480762 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:40.480428    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd" event={"ID":"059848e5-d957-47e3-b3f6-1343330bb458","Type":"ContainerStarted","Data":"4dcc0adb2668d8fae672b2689d99fb94a1f99d2d9a8d7d381babe61c4c0b045d"}
Jul 25 01:53:41.606440 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:41.606412    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"
Jul 25 01:53:41.611539 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:41.611520    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059848e5-d957-47e3-b3f6-1343330bb458-maas-api-service-ca\") pod \"059848e5-d957-47e3-b3f6-1343330bb458\" (UID: \"059848e5-d957-47e3-b3f6-1343330bb458\") "
Jul 25 01:53:41.611834 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:41.611815    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059848e5-d957-47e3-b3f6-1343330bb458-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "059848e5-d957-47e3-b3f6-1343330bb458" (UID: "059848e5-d957-47e3-b3f6-1343330bb458"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:53:41.712933 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:41.712897    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/059848e5-d957-47e3-b3f6-1343330bb458-maas-api-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:42.491028 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:42.491000    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"
Jul 25 01:53:42.491203 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:42.490999    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd" event={"ID":"059848e5-d957-47e3-b3f6-1343330bb458","Type":"ContainerDied","Data":"4dcc0adb2668d8fae672b2689d99fb94a1f99d2d9a8d7d381babe61c4c0b045d"}
Jul 25 01:53:42.491203 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:42.491125    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dcc0adb2668d8fae672b2689d99fb94a1f99d2d9a8d7d381babe61c4c0b045d"
Jul 25 01:53:43.872101 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:43.872061    2572 patch_prober.go:28] interesting pod/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Jul 25 01:53:43.872450 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:43.872128    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Jul 25 01:53:44.031418 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.031395    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:53:44.134765 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.134668    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-token\") pod \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") "
Jul 25 01:53:44.134765 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.134736    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istiod-ca-cert\") pod \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") "
Jul 25 01:53:44.135003 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.134786    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-data\") pod \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") "
Jul 25 01:53:44.135003 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.134809    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z7fj\" (UniqueName: \"kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-kube-api-access-5z7fj\") pod \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") "
Jul 25 01:53:44.135003 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.134839    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-socket\") pod \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") "
Jul 25 01:53:44.135003 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.134865    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-podinfo\") pod \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") "
Jul 25 01:53:44.135003 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.134891    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-certs\") pod \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") "
Jul 25 01:53:44.135003 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.134930    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-envoy\") pod \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") "
Jul 25 01:53:44.135003 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.135000    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-credential-socket\") pod \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\" (UID: \"57bf23a5-ce06-4842-a623-aa280b1ed9ae\") "
Jul 25 01:53:44.135384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.135148    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "57bf23a5-ce06-4842-a623-aa280b1ed9ae" (UID: "57bf23a5-ce06-4842-a623-aa280b1ed9ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:53:44.135384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.135172    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "57bf23a5-ce06-4842-a623-aa280b1ed9ae" (UID: "57bf23a5-ce06-4842-a623-aa280b1ed9ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:53:44.135384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.135241    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:44.135384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.135259    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istiod-ca-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:44.135384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.135355    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "57bf23a5-ce06-4842-a623-aa280b1ed9ae" (UID: "57bf23a5-ce06-4842-a623-aa280b1ed9ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:53:44.135640 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.135578    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "57bf23a5-ce06-4842-a623-aa280b1ed9ae" (UID: "57bf23a5-ce06-4842-a623-aa280b1ed9ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:53:44.135776 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.135754    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-data" (OuterVolumeSpecName: "istio-data") pod "57bf23a5-ce06-4842-a623-aa280b1ed9ae" (UID: "57bf23a5-ce06-4842-a623-aa280b1ed9ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:53:44.137283 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.137254    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-kube-api-access-5z7fj" (OuterVolumeSpecName: "kube-api-access-5z7fj") pod "57bf23a5-ce06-4842-a623-aa280b1ed9ae" (UID: "57bf23a5-ce06-4842-a623-aa280b1ed9ae"). InnerVolumeSpecName "kube-api-access-5z7fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:53:44.137397 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.137274    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "57bf23a5-ce06-4842-a623-aa280b1ed9ae" (UID: "57bf23a5-ce06-4842-a623-aa280b1ed9ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 25 01:53:44.137397 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.137298    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-token" (OuterVolumeSpecName: "istio-token") pod "57bf23a5-ce06-4842-a623-aa280b1ed9ae" (UID: "57bf23a5-ce06-4842-a623-aa280b1ed9ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:53:44.137397 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.137346    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "57bf23a5-ce06-4842-a623-aa280b1ed9ae" (UID: "57bf23a5-ce06-4842-a623-aa280b1ed9ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:53:44.236685 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.236647    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-credential-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:44.236685 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.236678    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-token\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:44.236685 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.236687    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-data\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:44.236919 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.236695    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z7fj\" (UniqueName: \"kubernetes.io/projected/57bf23a5-ce06-4842-a623-aa280b1ed9ae-kube-api-access-5z7fj\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:44.236919 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.236704    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-workload-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:44.236919 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.236712    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-podinfo\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:44.236919 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.236719    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57bf23a5-ce06-4842-a623-aa280b1ed9ae-istio-envoy\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:44.502384 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.502349    2572 generic.go:356] "Generic (PLEG): container finished" podID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerID="fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb" exitCode=0
Jul 25 01:53:44.502555 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.502393    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"
Jul 25 01:53:44.502555 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.502399    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" event={"ID":"57bf23a5-ce06-4842-a623-aa280b1ed9ae","Type":"ContainerDied","Data":"fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb"}
Jul 25 01:53:44.502555 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.502449    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s" event={"ID":"57bf23a5-ce06-4842-a623-aa280b1ed9ae","Type":"ContainerDied","Data":"b0ae7d568188ce0a171648670673c1959d27bbd39878ed0a096b63fc8df43231"}
Jul 25 01:53:44.502555 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.502473    2572 scope.go:117] "RemoveContainer" containerID="fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb"
Jul 25 01:53:44.514175 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.514147    2572 scope.go:117] "RemoveContainer" containerID="4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642"
Jul 25 01:53:44.520881 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.520856    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"]
Jul 25 01:53:44.523306 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.523282    2572 scope.go:117] "RemoveContainer" containerID="fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb"
Jul 25 01:53:44.523590 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:53:44.523567    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb\": container with ID starting with fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb not found: ID does not exist" containerID="fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb"
Jul 25 01:53:44.523688 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.523595    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb"} err="failed to get container status \"fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb\": rpc error: code = NotFound desc = could not find container \"fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb\": container with ID starting with fe619e6ced6142f29476cfc507c2497909cf3a867f745b0579e7fe12ce66eeeb not found: ID does not exist"
Jul 25 01:53:44.523688 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.523613    2572 scope.go:117] "RemoveContainer" containerID="4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642"
Jul 25 01:53:44.523857 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:53:44.523842    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642\": container with ID starting with 4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642 not found: ID does not exist" containerID="4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642"
Jul 25 01:53:44.523900 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.523865    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642"} err="failed to get container status \"4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642\": rpc error: code = NotFound desc = could not find container \"4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642\": container with ID starting with 4c54991cd7cb56c4827c4f9b16dbda806aa2a2b9ef68860308fd7e9359696642 not found: ID does not exist"
Jul 25 01:53:44.524126 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:44.524106    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-88f354-openshift-default-84c5d97dd8-96w8s"]
Jul 25 01:53:46.353069 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:46.353030    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" path="/var/lib/kubelet/pods/57bf23a5-ce06-4842-a623-aa280b1ed9ae/volumes"
Jul 25 01:53:54.675930 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:54.675897    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"]
Jul 25 01:53:54.676357 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:54.676102    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v" podUID="77f82d27-27a8-4df6-acea-3bec3116f608" containerName="maas-api" containerID="cri-o://032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f" gracePeriod=30
Jul 25 01:53:54.917040 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:54.917010    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:53:55.029107 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.029040    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkvjl\" (UniqueName: \"kubernetes.io/projected/77f82d27-27a8-4df6-acea-3bec3116f608-kube-api-access-xkvjl\") pod \"77f82d27-27a8-4df6-acea-3bec3116f608\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") "
Jul 25 01:53:55.029107 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.029102    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f82d27-27a8-4df6-acea-3bec3116f608-maas-api-tls\") pod \"77f82d27-27a8-4df6-acea-3bec3116f608\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") "
Jul 25 01:53:55.029296 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.029138    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f82d27-27a8-4df6-acea-3bec3116f608-openshift-service-ca\") pod \"77f82d27-27a8-4df6-acea-3bec3116f608\" (UID: \"77f82d27-27a8-4df6-acea-3bec3116f608\") "
Jul 25 01:53:55.029549 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.029519    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f82d27-27a8-4df6-acea-3bec3116f608-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77f82d27-27a8-4df6-acea-3bec3116f608" (UID: "77f82d27-27a8-4df6-acea-3bec3116f608"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 01:53:55.031024 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.030990    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f82d27-27a8-4df6-acea-3bec3116f608-kube-api-access-xkvjl" (OuterVolumeSpecName: "kube-api-access-xkvjl") pod "77f82d27-27a8-4df6-acea-3bec3116f608" (UID: "77f82d27-27a8-4df6-acea-3bec3116f608"). InnerVolumeSpecName "kube-api-access-xkvjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:53:55.031113 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.031088    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f82d27-27a8-4df6-acea-3bec3116f608-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77f82d27-27a8-4df6-acea-3bec3116f608" (UID: "77f82d27-27a8-4df6-acea-3bec3116f608"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:53:55.130642 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.130609    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkvjl\" (UniqueName: \"kubernetes.io/projected/77f82d27-27a8-4df6-acea-3bec3116f608-kube-api-access-xkvjl\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:55.130642 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.130636    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77f82d27-27a8-4df6-acea-3bec3116f608-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:55.130642 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.130646    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f82d27-27a8-4df6-acea-3bec3116f608-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:53:55.559033 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.558993    2572 generic.go:356] "Generic (PLEG): container finished" podID="77f82d27-27a8-4df6-acea-3bec3116f608" containerID="032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f" exitCode=0
Jul 25 01:53:55.559201 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.559043    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"
Jul 25 01:53:55.559201 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.559078    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v" event={"ID":"77f82d27-27a8-4df6-acea-3bec3116f608","Type":"ContainerDied","Data":"032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f"}
Jul 25 01:53:55.559201 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.559133    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v" event={"ID":"77f82d27-27a8-4df6-acea-3bec3116f608","Type":"ContainerDied","Data":"8438b7c78be7d2036c6349e6037dedd6190cb55aa0f8c486230bd97dcf29dd5f"}
Jul 25 01:53:55.559201 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.559156    2572 scope.go:117] "RemoveContainer" containerID="032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f"
Jul 25 01:53:55.568725 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.568708    2572 scope.go:117] "RemoveContainer" containerID="032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f"
Jul 25 01:53:55.568968 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:53:55.568949    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f\": container with ID starting with 032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f not found: ID does not exist" containerID="032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f"
Jul 25 01:53:55.569018 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.568977    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f"} err="failed to get container status \"032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f\": rpc error: code = NotFound desc = could not find container \"032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f\": container with ID starting with 032edc687a1185afed17852b0f4bc4c82a79cbb428ed7266ce8b3071d556c09f not found: ID does not exist"
Jul 25 01:53:55.580247 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.580219    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"]
Jul 25 01:53:55.583645 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:55.583626    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-393e3a-87b8d9cc7-d969v"]
Jul 25 01:53:56.352962 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:56.352932    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77f82d27-27a8-4df6-acea-3bec3116f608" path="/var/lib/kubelet/pods/77f82d27-27a8-4df6-acea-3bec3116f608/volumes"
Jul 25 01:53:59.238733 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:59.238696    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"]
Jul 25 01:53:59.243721 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:53:59.243695    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-393e3a-49vmd"]
Jul 25 01:54:00.353181 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:00.353143    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="059848e5-d957-47e3-b3f6-1343330bb458" path="/var/lib/kubelet/pods/059848e5-d957-47e3-b3f6-1343330bb458/volumes"
Jul 25 01:54:06.300407 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.300371    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"]
Jul 25 01:54:06.300866 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.300849    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy"
Jul 25 01:54:06.300940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.300868    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy"
Jul 25 01:54:06.300940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.300885    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="059848e5-d957-47e3-b3f6-1343330bb458" containerName="revoke-keys"
Jul 25 01:54:06.300940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.300894    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="059848e5-d957-47e3-b3f6-1343330bb458" containerName="revoke-keys"
Jul 25 01:54:06.300940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.300910    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy"
Jul 25 01:54:06.300940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.300919    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy"
Jul 25 01:54:06.300940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.300931    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77f82d27-27a8-4df6-acea-3bec3116f608" containerName="maas-api"
Jul 25 01:54:06.300940 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.300939    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="77f82d27-27a8-4df6-acea-3bec3116f608" containerName="maas-api"
Jul 25 01:54:06.301278 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.301030    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy"
Jul 25 01:54:06.301278 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.301045    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="059848e5-d957-47e3-b3f6-1343330bb458" containerName="revoke-keys"
Jul 25 01:54:06.301278 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.301055    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="57bf23a5-ce06-4842-a623-aa280b1ed9ae" containerName="istio-proxy"
Jul 25 01:54:06.301278 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.301065    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="77f82d27-27a8-4df6-acea-3bec3116f608" containerName="maas-api"
Jul 25 01:54:06.304051 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.304028    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.306367 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.306344    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 25 01:54:06.306499 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.306480    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-299af2-openshift-default-dockercfg-sgv6n\""
Jul 25 01:54:06.320614 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.320592    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"]
Jul 25 01:54:06.418752 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.418718    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-envoy\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.418928 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.418765    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-certs\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.418928 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.418818    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhsjk\" (UniqueName: \"kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-kube-api-access-fhsjk\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.419057 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.418956    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-socket\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.419057 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.418984    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istiod-ca-cert\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.419057 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.419042    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-podinfo\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.419210 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.419065    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-credential-socket\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.419210 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.419082    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-token\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.419316 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.419225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-data\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520377 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520345    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-socket\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520377 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520379    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istiod-ca-cert\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520583 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520411    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-podinfo\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520583 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520438    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-credential-socket\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520583 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520455    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-token\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520583 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520480    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-data\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520583 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520506    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-envoy\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520583 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520538    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-certs\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520583 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520575    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhsjk\" (UniqueName: \"kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-kube-api-access-fhsjk\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520944 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520761    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-socket\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520944 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520885    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-data\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.520944 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.520928    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-certs\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.521117 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.521087    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istiod-ca-cert\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.521178 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.521157    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-credential-socket\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.522966 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.522936    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-envoy\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.523070 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.523020    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-podinfo\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.528148 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.528125    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-token\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.528414 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.528395    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhsjk\" (UniqueName: \"kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-kube-api-access-fhsjk\") pod \"e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") " pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.615554 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.615486    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:06.746564 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.746534    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"]
Jul 25 01:54:06.748405 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:54:06.748375    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1388cc_c774_4532_8eea_04eaa8f7bba6.slice/crio-a581483abda701ab94d06678beba944812fc28c640bdd681f645888abbfb1350 WatchSource:0}: Error finding container a581483abda701ab94d06678beba944812fc28c640bdd681f645888abbfb1350: Status 404 returned error can't find the container with id a581483abda701ab94d06678beba944812fc28c640bdd681f645888abbfb1350
Jul 25 01:54:06.750753 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.750718    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:54:06.750839 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.750788    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:54:06.750884 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:06.750838    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:54:07.632147 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:07.632109    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" event={"ID":"ea1388cc-c774-4532-8eea-04eaa8f7bba6","Type":"ContainerStarted","Data":"0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a"}
Jul 25 01:54:07.632589 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:07.632165    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" event={"ID":"ea1388cc-c774-4532-8eea-04eaa8f7bba6","Type":"ContainerStarted","Data":"a581483abda701ab94d06678beba944812fc28c640bdd681f645888abbfb1350"}
Jul 25 01:54:07.653260 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:07.653203    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" podStartSLOduration=1.6531856660000002 podStartE2EDuration="1.653185666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:54:07.650525666 +0000 UTC m=+2531.760334950" watchObservedRunningTime="2026-07-25 01:54:07.653185666 +0000 UTC m=+2531.762994949"
Jul 25 01:54:08.616220 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:08.616182    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:08.621509 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:08.621474    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:08.636693 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:08.636668    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:08.637417 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:08.637397    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:09.645685 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.645647    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"]
Jul 25 01:54:09.649427 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.649403    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:09.651617 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.651597    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-299af2-serving-cert\""
Jul 25 01:54:09.651735 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.651647    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rhv77\""
Jul 25 01:54:09.660020 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.659990    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"]
Jul 25 01:54:09.750043 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.750004    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jbpq\" (UniqueName: \"kubernetes.io/projected/e1116a67-a2e0-4537-9868-e450307e7d32-kube-api-access-7jbpq\") pod \"maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:09.750234 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.750066    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1116a67-a2e0-4537-9868-e450307e7d32-maas-api-tls\") pod \"maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:09.750234 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.750127    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1116a67-a2e0-4537-9868-e450307e7d32-openshift-service-ca\") pod \"maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:09.850865 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.850832    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jbpq\" (UniqueName: \"kubernetes.io/projected/e1116a67-a2e0-4537-9868-e450307e7d32-kube-api-access-7jbpq\") pod \"maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:09.851039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.850895    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1116a67-a2e0-4537-9868-e450307e7d32-maas-api-tls\") pod \"maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:09.851039 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.850966    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1116a67-a2e0-4537-9868-e450307e7d32-openshift-service-ca\") pod \"maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:09.851782 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.851749    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1116a67-a2e0-4537-9868-e450307e7d32-openshift-service-ca\") pod \"maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:09.853749 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.853727    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1116a67-a2e0-4537-9868-e450307e7d32-maas-api-tls\") pod \"maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:09.865049 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.865023    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jbpq\" (UniqueName: \"kubernetes.io/projected/e1116a67-a2e0-4537-9868-e450307e7d32-kube-api-access-7jbpq\") pod \"maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:09.962787 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:09.962724    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:10.090879 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:10.090848    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"]
Jul 25 01:54:10.092434 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:54:10.092402    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1116a67_a2e0_4537_9868_e450307e7d32.slice/crio-06e43d15369c53fbb32e53156c4ee487f2a253d24219c5d018f28600dd03ce4d WatchSource:0}: Error finding container 06e43d15369c53fbb32e53156c4ee487f2a253d24219c5d018f28600dd03ce4d: Status 404 returned error can't find the container with id 06e43d15369c53fbb32e53156c4ee487f2a253d24219c5d018f28600dd03ce4d
Jul 25 01:54:10.648226 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:10.648185    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf" event={"ID":"e1116a67-a2e0-4537-9868-e450307e7d32","Type":"ContainerStarted","Data":"4aafe11c009886cff0d2fb2137126767e31d4b55300b884df65267111398285b"}
Jul 25 01:54:10.648738 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:10.648246    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf" event={"ID":"e1116a67-a2e0-4537-9868-e450307e7d32","Type":"ContainerStarted","Data":"06e43d15369c53fbb32e53156c4ee487f2a253d24219c5d018f28600dd03ce4d"}
Jul 25 01:54:10.648738 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:10.648501    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:10.665743 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:10.665695    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf" podStartSLOduration=1.3281363640000001 podStartE2EDuration="1.665678112s" totalImagesPullingTime="337.541748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:54:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:54:10.663672406 +0000 UTC m=+2534.773481713" watchObservedRunningTime="2026-07-25 01:54:10.665678112 +0000 UTC m=+2534.775487395"
Jul 25 01:54:16.659871 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:16.659839    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 01:54:18.184731 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.184697    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"]
Jul 25 01:54:18.188664 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.188640    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:18.190843 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.190820    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-ab0ae6-serving-cert\""
Jul 25 01:54:18.199380 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.199352    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"]
Jul 25 01:54:18.328301 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.328262    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgtsk\" (UniqueName: \"kubernetes.io/projected/80363b91-fbb0-458e-9241-d3120ee8b6e0-kube-api-access-jgtsk\") pod \"maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:18.328486 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.328349    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80363b91-fbb0-458e-9241-d3120ee8b6e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:18.328486 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.328400    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80363b91-fbb0-458e-9241-d3120ee8b6e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:18.429237 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.429205    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgtsk\" (UniqueName: \"kubernetes.io/projected/80363b91-fbb0-458e-9241-d3120ee8b6e0-kube-api-access-jgtsk\") pod \"maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:18.429418 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.429265    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80363b91-fbb0-458e-9241-d3120ee8b6e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:18.429418 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.429307    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80363b91-fbb0-458e-9241-d3120ee8b6e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:18.430036 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.430017    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80363b91-fbb0-458e-9241-d3120ee8b6e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:18.431664 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.431644    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80363b91-fbb0-458e-9241-d3120ee8b6e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:18.436964 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.436908    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgtsk\" (UniqueName: \"kubernetes.io/projected/80363b91-fbb0-458e-9241-d3120ee8b6e0-kube-api-access-jgtsk\") pod \"maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:18.502447 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.502420    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:18.637900 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.637871    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"]
Jul 25 01:54:18.639581 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:54:18.639551    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80363b91_fbb0_458e_9241_d3120ee8b6e0.slice/crio-b0d200e82c5a2e18fb9cf80c39b4158c22c302e102bb99691c65fce410ef93b3 WatchSource:0}: Error finding container b0d200e82c5a2e18fb9cf80c39b4158c22c302e102bb99691c65fce410ef93b3: Status 404 returned error can't find the container with id b0d200e82c5a2e18fb9cf80c39b4158c22c302e102bb99691c65fce410ef93b3
Jul 25 01:54:18.694349 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:18.694255    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf" event={"ID":"80363b91-fbb0-458e-9241-d3120ee8b6e0","Type":"ContainerStarted","Data":"b0d200e82c5a2e18fb9cf80c39b4158c22c302e102bb99691c65fce410ef93b3"}
Jul 25 01:54:19.702689 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:19.702649    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf" event={"ID":"80363b91-fbb0-458e-9241-d3120ee8b6e0","Type":"ContainerStarted","Data":"342b042817a2aa6676e4d36dca2c87658ab38c4aa4ea0f535c3ad93fb4d48966"}
Jul 25 01:54:19.703123 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:19.702756    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:19.719064 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:19.719000    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf" podStartSLOduration=1.381663091 podStartE2EDuration="1.718985522s" totalImagesPullingTime="337.322431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:54:19.718057949 +0000 UTC m=+2543.827867244" watchObservedRunningTime="2026-07-25 01:54:19.718985522 +0000 UTC m=+2543.828794842"
Jul 25 01:54:24.678528 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:24.678487    2572 patch_prober.go:28] interesting pod/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 25 01:54:24.678961 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:24.678559    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 25 01:54:25.714140 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:25.714110    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 01:54:26.741944 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:26.741908    2572 generic.go:356] "Generic (PLEG): container finished" podID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerID="0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a" exitCode=0
Jul 25 01:54:26.742398 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:26.741984    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" event={"ID":"ea1388cc-c774-4532-8eea-04eaa8f7bba6","Type":"ContainerDied","Data":"0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a"}
Jul 25 01:54:26.742398 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:26.742387    2572 scope.go:117] "RemoveContainer" containerID="0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a"
Jul 25 01:54:26.743379 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:26.743354    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:54:26.743484 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:26.743445    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:54:26.743548 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:26.743502    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 01:54:27.752411 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:27.752368    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" event={"ID":"ea1388cc-c774-4532-8eea-04eaa8f7bba6","Type":"ContainerStarted","Data":"8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c"}
Jul 25 01:54:28.615652 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:28.615612    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:28.615857 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:28.615823    2572 patch_prober.go:28] interesting pod/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 25 01:54:28.615946 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:28.615891    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 25 01:54:29.616436 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:29.616399    2572 patch_prober.go:28] interesting pod/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 25 01:54:29.616855 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:29.616454    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 25 01:54:30.616839 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:30.616800    2572 patch_prober.go:28] interesting pod/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Jul 25 01:54:30.617231 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:30.616874    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Jul 25 01:54:31.619470 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:31.619439    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:31.619845 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:31.619709    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:31.620412 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:31.620392    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 01:54:34.082291 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.082251    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"]
Jul 25 01:54:34.085592 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.085574    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.088687 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.088658    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"openshift-service-ca.crt\""
Jul 25 01:54:34.088833 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.088660    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"auth-test-model-299af2-kserve-self-signed-certs\""
Jul 25 01:54:34.088833 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.088662    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"default-dockercfg-hzcjs\""
Jul 25 01:54:34.088833 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.088735    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"kube-root-ca.crt\""
Jul 25 01:54:34.095718 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.095694    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"]
Jul 25 01:54:34.171913 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.171875    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-home\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.171913 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.171913    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-dshm\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.172151 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.171944    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195f599e-ae84-4ac4-bf01-153d60f70a36-tls-certs\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.172151 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.171970    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-model-cache\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.172151 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.172014    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv97p\" (UniqueName: \"kubernetes.io/projected/195f599e-ae84-4ac4-bf01-153d60f70a36-kube-api-access-cv97p\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.172151 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.172066    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-tmp-dir\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.273193 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.273159    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195f599e-ae84-4ac4-bf01-153d60f70a36-tls-certs\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.273386 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.273196    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-model-cache\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.273386 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.273227    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv97p\" (UniqueName: \"kubernetes.io/projected/195f599e-ae84-4ac4-bf01-153d60f70a36-kube-api-access-cv97p\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.273386 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.273246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-tmp-dir\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.273386 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.273306    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-home\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.273386 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.273319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-dshm\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.273910 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.273893    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-home\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.274006 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.273961    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-model-cache\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.274052 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.274003    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-tmp-dir\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.276194 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.276171    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-dshm\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.276364 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.276344    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195f599e-ae84-4ac4-bf01-153d60f70a36-tls-certs\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.280591 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.280574    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv97p\" (UniqueName: \"kubernetes.io/projected/195f599e-ae84-4ac4-bf01-153d60f70a36-kube-api-access-cv97p\") pod \"auth-test-model-299af2-kserve-56c95b944c-hlljb\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") " pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.397051 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.397020    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:34.528206 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.528180    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"]
Jul 25 01:54:34.530020 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:54:34.529994    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod195f599e_ae84_4ac4_bf01_153d60f70a36.slice/crio-deaad36a1d17b450a52747a8214f795f5f46253b06777217ba664ae54c8ca214 WatchSource:0}: Error finding container deaad36a1d17b450a52747a8214f795f5f46253b06777217ba664ae54c8ca214: Status 404 returned error can't find the container with id deaad36a1d17b450a52747a8214f795f5f46253b06777217ba664ae54c8ca214
Jul 25 01:54:34.803026 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.802928    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb" event={"ID":"195f599e-ae84-4ac4-bf01-153d60f70a36","Type":"ContainerStarted","Data":"815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65"}
Jul 25 01:54:34.803026 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.802977    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb" event={"ID":"195f599e-ae84-4ac4-bf01-153d60f70a36","Type":"ContainerStarted","Data":"deaad36a1d17b450a52747a8214f795f5f46253b06777217ba664ae54c8ca214"}
Jul 25 01:54:34.818837 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:34.818787    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb" podStartSLOduration=0.818771704 podStartE2EDuration="818.771704ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:54:34.817619338 +0000 UTC m=+2558.927428621" watchObservedRunningTime="2026-07-25 01:54:34.818771704 +0000 UTC m=+2558.928580988"
Jul 25 01:54:44.397616 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:44.397582    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:44.398134 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:44.397733    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:44.410152 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:44.410124    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:44.865450 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:44.865369    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:54:59.516219 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.516174    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"]
Jul 25 01:54:59.521035 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.521016    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.523749 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.523720    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"openshift-service-ca.crt\""
Jul 25 01:54:59.523873 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.523769    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"kube-root-ca.crt\""
Jul 25 01:54:59.525093 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.525070    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"default-dockercfg-fg9lv\""
Jul 25 01:54:59.525199 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.525105    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"auth-test-model-ab0ae6-kserve-self-signed-certs\""
Jul 25 01:54:59.531051 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.531032    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"]
Jul 25 01:54:59.685685 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.685645    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-tmp-dir\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.685883 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.685716    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-home\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.685883 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.685739    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-dshm\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.685883 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.685758    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a2c18c-af46-4d68-9797-5a762193b250-tls-certs\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.685883 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.685832    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbfnm\" (UniqueName: \"kubernetes.io/projected/08a2c18c-af46-4d68-9797-5a762193b250-kube-api-access-hbfnm\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.686042 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.685902    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-model-cache\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.787221 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.787133    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbfnm\" (UniqueName: \"kubernetes.io/projected/08a2c18c-af46-4d68-9797-5a762193b250-kube-api-access-hbfnm\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.787221 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.787175    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-model-cache\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.787221 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.787198    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-tmp-dir\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.787540 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.787237    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-home\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.787540 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.787253    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-dshm\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.787540 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.787267    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a2c18c-af46-4d68-9797-5a762193b250-tls-certs\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.787698 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.787642    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-model-cache\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.787698 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.787661    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-home\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.787775 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.787719    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-tmp-dir\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.789505 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.789483    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-dshm\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.789761 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.789743    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a2c18c-af46-4d68-9797-5a762193b250-tls-certs\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.797107 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.797088    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbfnm\" (UniqueName: \"kubernetes.io/projected/08a2c18c-af46-4d68-9797-5a762193b250-kube-api-access-hbfnm\") pod \"auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") " pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.830532 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.830509    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:54:59.987938 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:54:59.987902    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"]
Jul 25 01:54:59.992514 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:54:59.992486    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08a2c18c_af46_4d68_9797_5a762193b250.slice/crio-ef86311bbdf088fc689bee4872a9761678f8fe0d1b377cf8796a418ab60252a4 WatchSource:0}: Error finding container ef86311bbdf088fc689bee4872a9761678f8fe0d1b377cf8796a418ab60252a4: Status 404 returned error can't find the container with id ef86311bbdf088fc689bee4872a9761678f8fe0d1b377cf8796a418ab60252a4
Jul 25 01:55:00.937557 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:55:00.937527    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps" event={"ID":"08a2c18c-af46-4d68-9797-5a762193b250","Type":"ContainerStarted","Data":"c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96"}
Jul 25 01:55:00.937958 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:55:00.937570    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps" event={"ID":"08a2c18c-af46-4d68-9797-5a762193b250","Type":"ContainerStarted","Data":"ef86311bbdf088fc689bee4872a9761678f8fe0d1b377cf8796a418ab60252a4"}
Jul 25 01:55:00.959805 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:55:00.959758    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps" podStartSLOduration=1.9597419280000001 podStartE2EDuration="1.959741928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:55:00.957729529 +0000 UTC m=+2585.067538812" watchObservedRunningTime="2026-07-25 01:55:00.959741928 +0000 UTC m=+2585.069551212"
Jul 25 01:55:09.831234 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:55:09.831190    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:55:09.831689 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:55:09.831252    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:55:09.843834 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:55:09.843810    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:55:09.994222 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:55:09.994192    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:59:32.040585 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:32.040549    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"]
Jul 25 01:59:33.005948 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:33.005912    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"]
Jul 25 01:59:47.059310 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.059269    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb" podUID="195f599e-ae84-4ac4-bf01-153d60f70a36" containerName="main" containerID="cri-o://815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65" gracePeriod=45
Jul 25 01:59:47.309481 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.309413    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:59:47.375413 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.375382    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-dshm\") pod \"195f599e-ae84-4ac4-bf01-153d60f70a36\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") "
Jul 25 01:59:47.375413 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.375416    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-tmp-dir\") pod \"195f599e-ae84-4ac4-bf01-153d60f70a36\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") "
Jul 25 01:59:47.375672 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.375447    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195f599e-ae84-4ac4-bf01-153d60f70a36-tls-certs\") pod \"195f599e-ae84-4ac4-bf01-153d60f70a36\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") "
Jul 25 01:59:47.375672 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.375484    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-home\") pod \"195f599e-ae84-4ac4-bf01-153d60f70a36\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") "
Jul 25 01:59:47.375672 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.375528    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv97p\" (UniqueName: \"kubernetes.io/projected/195f599e-ae84-4ac4-bf01-153d60f70a36-kube-api-access-cv97p\") pod \"195f599e-ae84-4ac4-bf01-153d60f70a36\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") "
Jul 25 01:59:47.375672 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.375549    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-model-cache\") pod \"195f599e-ae84-4ac4-bf01-153d60f70a36\" (UID: \"195f599e-ae84-4ac4-bf01-153d60f70a36\") "
Jul 25 01:59:47.375879 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.375739    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "195f599e-ae84-4ac4-bf01-153d60f70a36" (UID: "195f599e-ae84-4ac4-bf01-153d60f70a36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:59:47.375879 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.375799    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-home" (OuterVolumeSpecName: "home") pod "195f599e-ae84-4ac4-bf01-153d60f70a36" (UID: "195f599e-ae84-4ac4-bf01-153d60f70a36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:59:47.375879 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.375837    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:47.375982 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.375888    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-model-cache" (OuterVolumeSpecName: "model-cache") pod "195f599e-ae84-4ac4-bf01-153d60f70a36" (UID: "195f599e-ae84-4ac4-bf01-153d60f70a36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:59:47.377711 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.377682    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-dshm" (OuterVolumeSpecName: "dshm") pod "195f599e-ae84-4ac4-bf01-153d60f70a36" (UID: "195f599e-ae84-4ac4-bf01-153d60f70a36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:59:47.377834 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.377819    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195f599e-ae84-4ac4-bf01-153d60f70a36-kube-api-access-cv97p" (OuterVolumeSpecName: "kube-api-access-cv97p") pod "195f599e-ae84-4ac4-bf01-153d60f70a36" (UID: "195f599e-ae84-4ac4-bf01-153d60f70a36"). InnerVolumeSpecName "kube-api-access-cv97p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:59:47.377834 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.377820    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195f599e-ae84-4ac4-bf01-153d60f70a36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "195f599e-ae84-4ac4-bf01-153d60f70a36" (UID: "195f599e-ae84-4ac4-bf01-153d60f70a36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:59:47.395753 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.395725    2572 generic.go:356] "Generic (PLEG): container finished" podID="195f599e-ae84-4ac4-bf01-153d60f70a36" containerID="815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65" exitCode=0
Jul 25 01:59:47.395861 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.395768    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"
Jul 25 01:59:47.395861 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.395805    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb" event={"ID":"195f599e-ae84-4ac4-bf01-153d60f70a36","Type":"ContainerDied","Data":"815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65"}
Jul 25 01:59:47.395861 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.395850    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb" event={"ID":"195f599e-ae84-4ac4-bf01-153d60f70a36","Type":"ContainerDied","Data":"deaad36a1d17b450a52747a8214f795f5f46253b06777217ba664ae54c8ca214"}
Jul 25 01:59:47.395995 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.395871    2572 scope.go:117] "RemoveContainer" containerID="815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65"
Jul 25 01:59:47.405155 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.405139    2572 scope.go:117] "RemoveContainer" containerID="815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65"
Jul 25 01:59:47.405445 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:59:47.405426    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65\": container with ID starting with 815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65 not found: ID does not exist" containerID="815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65"
Jul 25 01:59:47.405503 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.405454    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65"} err="failed to get container status \"815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65\": rpc error: code = NotFound desc = could not find container \"815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65\": container with ID starting with 815ed314ce777f4e6a7080bf4a0a30c1cc5886891b8d28f1bddc8ca4ccf7cd65 not found: ID does not exist"
Jul 25 01:59:47.416068 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.416038    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"]
Jul 25 01:59:47.421541 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.421519    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/auth-test-model-299af2-kserve-56c95b944c-hlljb"]
Jul 25 01:59:47.476472 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.476440    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:47.476472 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.476466    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv97p\" (UniqueName: \"kubernetes.io/projected/195f599e-ae84-4ac4-bf01-153d60f70a36-kube-api-access-cv97p\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:47.476472 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.476475    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:47.476692 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.476484    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/195f599e-ae84-4ac4-bf01-153d60f70a36-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:47.476692 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:47.476494    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/195f599e-ae84-4ac4-bf01-153d60f70a36-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:48.025363 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.025306    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps" podUID="08a2c18c-af46-4d68-9797-5a762193b250" containerName="main" containerID="cri-o://c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96" gracePeriod=45
Jul 25 01:59:48.284411 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.284347    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:59:48.353133 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.353100    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195f599e-ae84-4ac4-bf01-153d60f70a36" path="/var/lib/kubelet/pods/195f599e-ae84-4ac4-bf01-153d60f70a36/volumes"
Jul 25 01:59:48.385027 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.384994    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbfnm\" (UniqueName: \"kubernetes.io/projected/08a2c18c-af46-4d68-9797-5a762193b250-kube-api-access-hbfnm\") pod \"08a2c18c-af46-4d68-9797-5a762193b250\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") "
Jul 25 01:59:48.385195 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.385032    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-model-cache\") pod \"08a2c18c-af46-4d68-9797-5a762193b250\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") "
Jul 25 01:59:48.385195 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.385058    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-home\") pod \"08a2c18c-af46-4d68-9797-5a762193b250\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") "
Jul 25 01:59:48.385195 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.385092    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-dshm\") pod \"08a2c18c-af46-4d68-9797-5a762193b250\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") "
Jul 25 01:59:48.385195 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.385119    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-tmp-dir\") pod \"08a2c18c-af46-4d68-9797-5a762193b250\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") "
Jul 25 01:59:48.385195 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.385166    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a2c18c-af46-4d68-9797-5a762193b250-tls-certs\") pod \"08a2c18c-af46-4d68-9797-5a762193b250\" (UID: \"08a2c18c-af46-4d68-9797-5a762193b250\") "
Jul 25 01:59:48.385486 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.385375    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-model-cache" (OuterVolumeSpecName: "model-cache") pod "08a2c18c-af46-4d68-9797-5a762193b250" (UID: "08a2c18c-af46-4d68-9797-5a762193b250"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:59:48.385486 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.385417    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-home" (OuterVolumeSpecName: "home") pod "08a2c18c-af46-4d68-9797-5a762193b250" (UID: "08a2c18c-af46-4d68-9797-5a762193b250"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:59:48.385486 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.385433    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08a2c18c-af46-4d68-9797-5a762193b250" (UID: "08a2c18c-af46-4d68-9797-5a762193b250"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:59:48.387374 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.387346    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a2c18c-af46-4d68-9797-5a762193b250-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08a2c18c-af46-4d68-9797-5a762193b250" (UID: "08a2c18c-af46-4d68-9797-5a762193b250"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 01:59:48.387471 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.387369    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-dshm" (OuterVolumeSpecName: "dshm") pod "08a2c18c-af46-4d68-9797-5a762193b250" (UID: "08a2c18c-af46-4d68-9797-5a762193b250"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 01:59:48.387471 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.387383    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a2c18c-af46-4d68-9797-5a762193b250-kube-api-access-hbfnm" (OuterVolumeSpecName: "kube-api-access-hbfnm") pod "08a2c18c-af46-4d68-9797-5a762193b250" (UID: "08a2c18c-af46-4d68-9797-5a762193b250"). InnerVolumeSpecName "kube-api-access-hbfnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 01:59:48.404365 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.404338    2572 generic.go:356] "Generic (PLEG): container finished" podID="08a2c18c-af46-4d68-9797-5a762193b250" containerID="c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96" exitCode=0
Jul 25 01:59:48.404482 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.404368    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps" event={"ID":"08a2c18c-af46-4d68-9797-5a762193b250","Type":"ContainerDied","Data":"c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96"}
Jul 25 01:59:48.404482 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.404391    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"
Jul 25 01:59:48.404482 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.404408    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps" event={"ID":"08a2c18c-af46-4d68-9797-5a762193b250","Type":"ContainerDied","Data":"ef86311bbdf088fc689bee4872a9761678f8fe0d1b377cf8796a418ab60252a4"}
Jul 25 01:59:48.404482 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.404430    2572 scope.go:117] "RemoveContainer" containerID="c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96"
Jul 25 01:59:48.412949 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.412930    2572 scope.go:117] "RemoveContainer" containerID="c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96"
Jul 25 01:59:48.413197 ip-10-0-135-99 kubenswrapper[2572]: E0725 01:59:48.413180    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96\": container with ID starting with c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96 not found: ID does not exist" containerID="c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96"
Jul 25 01:59:48.413249 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.413202    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96"} err="failed to get container status \"c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96\": rpc error: code = NotFound desc = could not find container \"c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96\": container with ID starting with c7e4396cc3b1ccb28ae9f037c09ec857ec25ac28bfb4c48e746e115e124eac96 not found: ID does not exist"
Jul 25 01:59:48.425916 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.425896    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"]
Jul 25 01:59:48.427823 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.427806    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/auth-test-model-ab0ae6-kserve-dd5f49bf9-rdqps"]
Jul 25 01:59:48.486186 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.486155    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a2c18c-af46-4d68-9797-5a762193b250-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:48.486186 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.486182    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbfnm\" (UniqueName: \"kubernetes.io/projected/08a2c18c-af46-4d68-9797-5a762193b250-kube-api-access-hbfnm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:48.486186 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.486193    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:48.486186 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.486203    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:48.486492 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.486212    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:48.486492 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:48.486220    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a2c18c-af46-4d68-9797-5a762193b250-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 01:59:50.354315 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:50.354279    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a2c18c-af46-4d68-9797-5a762193b250" path="/var/lib/kubelet/pods/08a2c18c-af46-4d68-9797-5a762193b250/volumes"
Jul 25 01:59:56.602909 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.602871    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"]
Jul 25 01:59:56.603411 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.603393    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195f599e-ae84-4ac4-bf01-153d60f70a36" containerName="main"
Jul 25 01:59:56.603461 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.603414    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="195f599e-ae84-4ac4-bf01-153d60f70a36" containerName="main"
Jul 25 01:59:56.603461 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.603429    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a2c18c-af46-4d68-9797-5a762193b250" containerName="main"
Jul 25 01:59:56.603461 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.603440    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a2c18c-af46-4d68-9797-5a762193b250" containerName="main"
Jul 25 01:59:56.603559 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.603537    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="195f599e-ae84-4ac4-bf01-153d60f70a36" containerName="main"
Jul 25 01:59:56.603559 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.603551    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a2c18c-af46-4d68-9797-5a762193b250" containerName="main"
Jul 25 01:59:56.608218 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.608198    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.610516 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.610483    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"sub-test-model-299af2-kserve-self-signed-certs\""
Jul 25 01:59:56.610516 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.610493    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"kube-root-ca.crt\""
Jul 25 01:59:56.610712 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.610516    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"openshift-service-ca.crt\""
Jul 25 01:59:56.610712 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.610594    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"default-dockercfg-hzcjs\""
Jul 25 01:59:56.614776 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.614753    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"]
Jul 25 01:59:56.739837 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.739803    2572 scope.go:117] "RemoveContainer" containerID="7f42f771df13ca6a39f4c672f6bba02901e2b5e0d5c1470a472d816875261e1a"
Jul 25 01:59:56.748093 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.748068    2572 scope.go:117] "RemoveContainer" containerID="3d7f732f5ce04db993d871e389c2f7bf6a43c3bfd99c610858036e8c792eb4d9"
Jul 25 01:59:56.760572 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.760551    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-model-cache\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.760628 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.760604    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tls-certs\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.760697 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.760638    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfcn\" (UniqueName: \"kubernetes.io/projected/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-kube-api-access-pkfcn\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.760697 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.760672    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-dshm\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.760778 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.760705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-home\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.760778 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.760720    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tmp-dir\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.861512 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.861411    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tmp-dir\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.861512 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.861470    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-model-cache\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.861750 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.861578    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tls-certs\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.861750 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.861614    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfcn\" (UniqueName: \"kubernetes.io/projected/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-kube-api-access-pkfcn\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.861750 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.861642    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-dshm\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.861750 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.861688    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-home\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.861964 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.861877    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-model-cache\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.861964 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.861914    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tmp-dir\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.862036 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.861963    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-home\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.863905 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.863882    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-dshm\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.864032 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.864016    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tls-certs\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.868954 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.868931    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfcn\" (UniqueName: \"kubernetes.io/projected/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-kube-api-access-pkfcn\") pod \"sub-test-model-299af2-kserve-5db6499665-bgrs7\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") " pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:56.919630 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:56.919595    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 01:59:57.051744 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:57.051715    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"]
Jul 25 01:59:57.053112 ip-10-0-135-99 kubenswrapper[2572]: W0725 01:59:57.053089    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6506b033_6c94_4bbf_8cb5_9dcd205db9a5.slice/crio-8ba09b30e5e3bc589fcdba1dc3e6bdc3d3276965e65b8dc64cc98c32f2060fe1 WatchSource:0}: Error finding container 8ba09b30e5e3bc589fcdba1dc3e6bdc3d3276965e65b8dc64cc98c32f2060fe1: Status 404 returned error can't find the container with id 8ba09b30e5e3bc589fcdba1dc3e6bdc3d3276965e65b8dc64cc98c32f2060fe1
Jul 25 01:59:57.458719 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:57.458683    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7" event={"ID":"6506b033-6c94-4bbf-8cb5-9dcd205db9a5","Type":"ContainerStarted","Data":"7e0be7b50cfac408a4af0cc861d797cf954d95cb1a09be757e8f1b53ae23b11a"}
Jul 25 01:59:57.458719 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:57.458721    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7" event={"ID":"6506b033-6c94-4bbf-8cb5-9dcd205db9a5","Type":"ContainerStarted","Data":"8ba09b30e5e3bc589fcdba1dc3e6bdc3d3276965e65b8dc64cc98c32f2060fe1"}
Jul 25 01:59:57.474925 ip-10-0-135-99 kubenswrapper[2572]: I0725 01:59:57.474868    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7" podStartSLOduration=1.474851513 podStartE2EDuration="1.474851513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 01:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 01:59:57.473484032 +0000 UTC m=+2881.583293317" watchObservedRunningTime="2026-07-25 01:59:57.474851513 +0000 UTC m=+2881.584660797"
Jul 25 02:00:00.130776 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.130736    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth"]
Jul 25 02:00:00.134187 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.134168    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29749080-x4lsk"]
Jul 25 02:00:00.134310 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.134289    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth"
Jul 25 02:00:00.136484 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.136463    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 02:00:00.137499 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.137481    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749080-x4lsk"
Jul 25 02:00:00.141896 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.141858    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth"]
Jul 25 02:00:00.145201 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.145174    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29749080-x4lsk"]
Jul 25 02:00:00.149447 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.149417    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth"
Jul 25 02:00:00.156679 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.156659    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749080-x4lsk"
Jul 25 02:00:00.292289 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.292256    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth"]
Jul 25 02:00:00.294553 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:00:00.294511    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57da19f1_6349_4ef4_b6fc_072011bccbba.slice/crio-de15a16d48c7f40f1d263ac0c9b4fae7de764b856e6930b24f8f758d8f9d2c24 WatchSource:0}: Error finding container de15a16d48c7f40f1d263ac0c9b4fae7de764b856e6930b24f8f758d8f9d2c24: Status 404 returned error can't find the container with id de15a16d48c7f40f1d263ac0c9b4fae7de764b856e6930b24f8f758d8f9d2c24
Jul 25 02:00:00.319081 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.319053    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29749080-x4lsk"]
Jul 25 02:00:00.320491 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:00:00.320454    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ddb903_a328_4b87_85a1_cbaa539b16da.slice/crio-7cb2dd6e219fcf9e27adf4500e6d0811d5835322265e04654429e27fd2b70958 WatchSource:0}: Error finding container 7cb2dd6e219fcf9e27adf4500e6d0811d5835322265e04654429e27fd2b70958: Status 404 returned error can't find the container with id 7cb2dd6e219fcf9e27adf4500e6d0811d5835322265e04654429e27fd2b70958
Jul 25 02:00:00.476405 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.476370    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth" event={"ID":"57da19f1-6349-4ef4-b6fc-072011bccbba","Type":"ContainerStarted","Data":"f782dc70d51867a6367b271da3a817f854116de4a8b997c1f4abe8e88128d83d"}
Jul 25 02:00:00.476576 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.476437    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth" event={"ID":"57da19f1-6349-4ef4-b6fc-072011bccbba","Type":"ContainerStarted","Data":"de15a16d48c7f40f1d263ac0c9b4fae7de764b856e6930b24f8f758d8f9d2c24"}
Jul 25 02:00:00.477498 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.477473    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749080-x4lsk" event={"ID":"23ddb903-a328-4b87-85a1-cbaa539b16da","Type":"ContainerStarted","Data":"7cb2dd6e219fcf9e27adf4500e6d0811d5835322265e04654429e27fd2b70958"}
Jul 25 02:00:00.490772 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:00.490726    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth" podStartSLOduration=0.490712202 podStartE2EDuration="490.712202ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:00:00.489714164 +0000 UTC m=+2884.599523450" watchObservedRunningTime="2026-07-25 02:00:00.490712202 +0000 UTC m=+2884.600521485"
Jul 25 02:00:01.484396 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:01.484362    2572 generic.go:356] "Generic (PLEG): container finished" podID="57da19f1-6349-4ef4-b6fc-072011bccbba" containerID="f782dc70d51867a6367b271da3a817f854116de4a8b997c1f4abe8e88128d83d" exitCode=0
Jul 25 02:00:01.484837 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:01.484455    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth" event={"ID":"57da19f1-6349-4ef4-b6fc-072011bccbba","Type":"ContainerDied","Data":"f782dc70d51867a6367b271da3a817f854116de4a8b997c1f4abe8e88128d83d"}
Jul 25 02:00:01.485822 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:01.485795    2572 generic.go:356] "Generic (PLEG): container finished" podID="23ddb903-a328-4b87-85a1-cbaa539b16da" containerID="b9457b31ff6342a46ff54dc0cd48ec16330f5d06e01f52a98d4779dedc38b6cb" exitCode=0
Jul 25 02:00:01.485967 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:01.485824    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749080-x4lsk" event={"ID":"23ddb903-a328-4b87-85a1-cbaa539b16da","Type":"ContainerDied","Data":"b9457b31ff6342a46ff54dc0cd48ec16330f5d06e01f52a98d4779dedc38b6cb"}
Jul 25 02:00:02.623773 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:02.623743    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749080-x4lsk"
Jul 25 02:00:02.657911 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:02.657881    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth"
Jul 25 02:00:03.496910 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:03.496878    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth"
Jul 25 02:00:03.497114 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:03.496879    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth" event={"ID":"57da19f1-6349-4ef4-b6fc-072011bccbba","Type":"ContainerDied","Data":"de15a16d48c7f40f1d263ac0c9b4fae7de764b856e6930b24f8f758d8f9d2c24"}
Jul 25 02:00:03.497114 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:03.497022    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de15a16d48c7f40f1d263ac0c9b4fae7de764b856e6930b24f8f758d8f9d2c24"
Jul 25 02:00:03.498295 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:03.498273    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749080-x4lsk"
Jul 25 02:00:03.498439 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:03.498356    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29749080-x4lsk" event={"ID":"23ddb903-a328-4b87-85a1-cbaa539b16da","Type":"ContainerDied","Data":"7cb2dd6e219fcf9e27adf4500e6d0811d5835322265e04654429e27fd2b70958"}
Jul 25 02:00:03.498439 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:03.498404    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cb2dd6e219fcf9e27adf4500e6d0811d5835322265e04654429e27fd2b70958"
Jul 25 02:00:06.920377 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:06.920345    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 02:00:06.920880 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:06.920400    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 02:00:06.932750 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:06.932725    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 02:00:07.531206 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:07.531174    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 02:00:28.893478 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.893440    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"]
Jul 25 02:00:28.893915 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.893798    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ddb903-a328-4b87-85a1-cbaa539b16da" containerName="cleanup"
Jul 25 02:00:28.893915 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.893813    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ddb903-a328-4b87-85a1-cbaa539b16da" containerName="cleanup"
Jul 25 02:00:28.893915 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.893826    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57da19f1-6349-4ef4-b6fc-072011bccbba" containerName="cleanup"
Jul 25 02:00:28.893915 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.893832    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="57da19f1-6349-4ef4-b6fc-072011bccbba" containerName="cleanup"
Jul 25 02:00:28.893915 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.893899    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ddb903-a328-4b87-85a1-cbaa539b16da" containerName="cleanup"
Jul 25 02:00:28.893915 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.893910    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="57da19f1-6349-4ef4-b6fc-072011bccbba" containerName="cleanup"
Jul 25 02:00:28.896982 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.896964    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:28.904108 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.900100    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"openshift-service-ca.crt\""
Jul 25 02:00:28.904108 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.900591    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"sub-test-model-ab0ae6-kserve-self-signed-certs\""
Jul 25 02:00:28.904108 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.900674    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"default-dockercfg-fg9lv\""
Jul 25 02:00:28.904108 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.900939    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"kube-root-ca.crt\""
Jul 25 02:00:28.911959 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.911924    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"]
Jul 25 02:00:28.946782 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.946743    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-dshm\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:28.946989 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.946813    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-model-cache\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:28.946989 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.946925    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-home\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:28.946989 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.946972    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2eb3a36-b369-446f-af09-46a760c1be05-tls-certs\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:28.947138 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.946999    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-tmp-dir\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:28.947138 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:28.947023    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmn7\" (UniqueName: \"kubernetes.io/projected/f2eb3a36-b369-446f-af09-46a760c1be05-kube-api-access-lqmn7\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.047897 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.047859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-home\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.048087 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.047904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2eb3a36-b369-446f-af09-46a760c1be05-tls-certs\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.048087 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.047932    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-tmp-dir\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.048087 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.047957    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmn7\" (UniqueName: \"kubernetes.io/projected/f2eb3a36-b369-446f-af09-46a760c1be05-kube-api-access-lqmn7\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.048087 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.047987    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-dshm\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.048087 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.048070    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-model-cache\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.048442 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.048368    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-home\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.048442 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.048424    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-tmp-dir\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.048540 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.048523    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-model-cache\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.050553 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.050523    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-dshm\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.050698 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.050681    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2eb3a36-b369-446f-af09-46a760c1be05-tls-certs\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.055504 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.055481    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmn7\" (UniqueName: \"kubernetes.io/projected/f2eb3a36-b369-446f-af09-46a760c1be05-kube-api-access-lqmn7\") pod \"sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") " pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.213709 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.213615    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:29.350415 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.350389    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"]
Jul 25 02:00:29.352058 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:00:29.352030    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2eb3a36_b369_446f_af09_46a760c1be05.slice/crio-6caa256658afb5a29f29f188e6217d579c442e696f20c4bd112969230fd3901f WatchSource:0}: Error finding container 6caa256658afb5a29f29f188e6217d579c442e696f20c4bd112969230fd3901f: Status 404 returned error can't find the container with id 6caa256658afb5a29f29f188e6217d579c442e696f20c4bd112969230fd3901f
Jul 25 02:00:29.630873 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.630777    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq" event={"ID":"f2eb3a36-b369-446f-af09-46a760c1be05","Type":"ContainerStarted","Data":"3859ed123b4e8863d2633cb1e01ff435119fecef706fe67129c2be9c29d18d85"}
Jul 25 02:00:29.630873 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.630823    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq" event={"ID":"f2eb3a36-b369-446f-af09-46a760c1be05","Type":"ContainerStarted","Data":"6caa256658afb5a29f29f188e6217d579c442e696f20c4bd112969230fd3901f"}
Jul 25 02:00:29.647933 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:29.647880    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq" podStartSLOduration=1.647864564 podStartE2EDuration="1.647864564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:00:29.646737756 +0000 UTC m=+2913.756547051" watchObservedRunningTime="2026-07-25 02:00:29.647864564 +0000 UTC m=+2913.757673847"
Jul 25 02:00:39.214004 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:39.213964    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:39.214570 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:39.214035    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:39.226318 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:39.226295    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:00:39.694050 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:00:39.694020    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:01:58.006883 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:01:58.006846    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"]
Jul 25 02:01:58.997224 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:01:58.997191    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"]
Jul 25 02:02:13.025745 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.025683    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7" podUID="6506b033-6c94-4bbf-8cb5-9dcd205db9a5" containerName="main" containerID="cri-o://7e0be7b50cfac408a4af0cc861d797cf954d95cb1a09be757e8f1b53ae23b11a" gracePeriod=45
Jul 25 02:02:13.162620 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.162586    2572 generic.go:356] "Generic (PLEG): container finished" podID="6506b033-6c94-4bbf-8cb5-9dcd205db9a5" containerID="7e0be7b50cfac408a4af0cc861d797cf954d95cb1a09be757e8f1b53ae23b11a" exitCode=0
Jul 25 02:02:13.162620 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.162611    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7" event={"ID":"6506b033-6c94-4bbf-8cb5-9dcd205db9a5","Type":"ContainerDied","Data":"7e0be7b50cfac408a4af0cc861d797cf954d95cb1a09be757e8f1b53ae23b11a"}
Jul 25 02:02:13.278990 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.278919    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 02:02:13.366980 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.366948    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkfcn\" (UniqueName: \"kubernetes.io/projected/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-kube-api-access-pkfcn\") pod \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") "
Jul 25 02:02:13.367173 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.366991    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tmp-dir\") pod \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") "
Jul 25 02:02:13.367173 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.367040    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-model-cache\") pod \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") "
Jul 25 02:02:13.367173 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.367057    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-home\") pod \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") "
Jul 25 02:02:13.367173 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.367138    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-dshm\") pod \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") "
Jul 25 02:02:13.367173 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.367167    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tls-certs\") pod \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\" (UID: \"6506b033-6c94-4bbf-8cb5-9dcd205db9a5\") "
Jul 25 02:02:13.367483 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.367372    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6506b033-6c94-4bbf-8cb5-9dcd205db9a5" (UID: "6506b033-6c94-4bbf-8cb5-9dcd205db9a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:02:13.367483 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.367389    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "6506b033-6c94-4bbf-8cb5-9dcd205db9a5" (UID: "6506b033-6c94-4bbf-8cb5-9dcd205db9a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:02:13.367483 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.367461    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-home" (OuterVolumeSpecName: "home") pod "6506b033-6c94-4bbf-8cb5-9dcd205db9a5" (UID: "6506b033-6c94-4bbf-8cb5-9dcd205db9a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:02:13.367619 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.367512    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:13.367619 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.367533    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:13.369191 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.369166    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-dshm" (OuterVolumeSpecName: "dshm") pod "6506b033-6c94-4bbf-8cb5-9dcd205db9a5" (UID: "6506b033-6c94-4bbf-8cb5-9dcd205db9a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:02:13.369305 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.369229    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-kube-api-access-pkfcn" (OuterVolumeSpecName: "kube-api-access-pkfcn") pod "6506b033-6c94-4bbf-8cb5-9dcd205db9a5" (UID: "6506b033-6c94-4bbf-8cb5-9dcd205db9a5"). InnerVolumeSpecName "kube-api-access-pkfcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:02:13.369305 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.369229    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6506b033-6c94-4bbf-8cb5-9dcd205db9a5" (UID: "6506b033-6c94-4bbf-8cb5-9dcd205db9a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 02:02:13.468213 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.468163    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkfcn\" (UniqueName: \"kubernetes.io/projected/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-kube-api-access-pkfcn\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:13.468213 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.468208    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:13.468213 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.468218    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:13.468213 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:13.468227    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6506b033-6c94-4bbf-8cb5-9dcd205db9a5-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:14.016120 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.016055    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq" podUID="f2eb3a36-b369-446f-af09-46a760c1be05" containerName="main" containerID="cri-o://3859ed123b4e8863d2633cb1e01ff435119fecef706fe67129c2be9c29d18d85" gracePeriod=45
Jul 25 02:02:14.169923 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.169885    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7" event={"ID":"6506b033-6c94-4bbf-8cb5-9dcd205db9a5","Type":"ContainerDied","Data":"8ba09b30e5e3bc589fcdba1dc3e6bdc3d3276965e65b8dc64cc98c32f2060fe1"}
Jul 25 02:02:14.170383 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.169945    2572 scope.go:117] "RemoveContainer" containerID="7e0be7b50cfac408a4af0cc861d797cf954d95cb1a09be757e8f1b53ae23b11a"
Jul 25 02:02:14.170383 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.169895    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"
Jul 25 02:02:14.171676 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.171647    2572 generic.go:356] "Generic (PLEG): container finished" podID="f2eb3a36-b369-446f-af09-46a760c1be05" containerID="3859ed123b4e8863d2633cb1e01ff435119fecef706fe67129c2be9c29d18d85" exitCode=0
Jul 25 02:02:14.171794 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.171685    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq" event={"ID":"f2eb3a36-b369-446f-af09-46a760c1be05","Type":"ContainerDied","Data":"3859ed123b4e8863d2633cb1e01ff435119fecef706fe67129c2be9c29d18d85"}
Jul 25 02:02:14.194856 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.194823    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"]
Jul 25 02:02:14.196992 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.196966    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/sub-test-model-299af2-kserve-5db6499665-bgrs7"]
Jul 25 02:02:14.277913 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.277851    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:02:14.353094 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.353057    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6506b033-6c94-4bbf-8cb5-9dcd205db9a5" path="/var/lib/kubelet/pods/6506b033-6c94-4bbf-8cb5-9dcd205db9a5/volumes"
Jul 25 02:02:14.374170 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374145    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-home\") pod \"f2eb3a36-b369-446f-af09-46a760c1be05\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") "
Jul 25 02:02:14.374309 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374201    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-model-cache\") pod \"f2eb3a36-b369-446f-af09-46a760c1be05\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") "
Jul 25 02:02:14.374309 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374225    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqmn7\" (UniqueName: \"kubernetes.io/projected/f2eb3a36-b369-446f-af09-46a760c1be05-kube-api-access-lqmn7\") pod \"f2eb3a36-b369-446f-af09-46a760c1be05\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") "
Jul 25 02:02:14.374309 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374279    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2eb3a36-b369-446f-af09-46a760c1be05-tls-certs\") pod \"f2eb3a36-b369-446f-af09-46a760c1be05\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") "
Jul 25 02:02:14.374309 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374294    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-tmp-dir\") pod \"f2eb3a36-b369-446f-af09-46a760c1be05\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") "
Jul 25 02:02:14.374562 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374314    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-dshm\") pod \"f2eb3a36-b369-446f-af09-46a760c1be05\" (UID: \"f2eb3a36-b369-446f-af09-46a760c1be05\") "
Jul 25 02:02:14.374562 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374459    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-home" (OuterVolumeSpecName: "home") pod "f2eb3a36-b369-446f-af09-46a760c1be05" (UID: "f2eb3a36-b369-446f-af09-46a760c1be05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:02:14.374562 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374546    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2eb3a36-b369-446f-af09-46a760c1be05" (UID: "f2eb3a36-b369-446f-af09-46a760c1be05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:02:14.374687 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374563    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2eb3a36-b369-446f-af09-46a760c1be05" (UID: "f2eb3a36-b369-446f-af09-46a760c1be05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:02:14.374725 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374691    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:14.374725 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374705    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:14.374725 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.374717    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:14.376399 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.376380    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-dshm" (OuterVolumeSpecName: "dshm") pod "f2eb3a36-b369-446f-af09-46a760c1be05" (UID: "f2eb3a36-b369-446f-af09-46a760c1be05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:02:14.376473 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.376424    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2eb3a36-b369-446f-af09-46a760c1be05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2eb3a36-b369-446f-af09-46a760c1be05" (UID: "f2eb3a36-b369-446f-af09-46a760c1be05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 02:02:14.376528 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.376509    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2eb3a36-b369-446f-af09-46a760c1be05-kube-api-access-lqmn7" (OuterVolumeSpecName: "kube-api-access-lqmn7") pod "f2eb3a36-b369-446f-af09-46a760c1be05" (UID: "f2eb3a36-b369-446f-af09-46a760c1be05"). InnerVolumeSpecName "kube-api-access-lqmn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:02:14.476180 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.476142    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmn7\" (UniqueName: \"kubernetes.io/projected/f2eb3a36-b369-446f-af09-46a760c1be05-kube-api-access-lqmn7\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:14.476180 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.476173    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2eb3a36-b369-446f-af09-46a760c1be05-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:14.476180 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:14.476184    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2eb3a36-b369-446f-af09-46a760c1be05-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:02:15.179686 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:15.179652    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"
Jul 25 02:02:15.180126 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:15.179690    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq" event={"ID":"f2eb3a36-b369-446f-af09-46a760c1be05","Type":"ContainerDied","Data":"6caa256658afb5a29f29f188e6217d579c442e696f20c4bd112969230fd3901f"}
Jul 25 02:02:15.180126 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:15.179739    2572 scope.go:117] "RemoveContainer" containerID="3859ed123b4e8863d2633cb1e01ff435119fecef706fe67129c2be9c29d18d85"
Jul 25 02:02:15.200854 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:15.200827    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"]
Jul 25 02:02:15.202799 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:15.202778    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/sub-test-model-ab0ae6-kserve-8655c6f97-nmsbq"]
Jul 25 02:02:16.352782 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:16.352742    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2eb3a36-b369-446f-af09-46a760c1be05" path="/var/lib/kubelet/pods/f2eb3a36-b369-446f-af09-46a760c1be05/volumes"
Jul 25 02:02:22.593786 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.593756    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"]
Jul 25 02:02:22.594263 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.594080    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6506b033-6c94-4bbf-8cb5-9dcd205db9a5" containerName="main"
Jul 25 02:02:22.594263 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.594091    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6506b033-6c94-4bbf-8cb5-9dcd205db9a5" containerName="main"
Jul 25 02:02:22.594263 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.594105    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2eb3a36-b369-446f-af09-46a760c1be05" containerName="main"
Jul 25 02:02:22.594263 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.594111    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2eb3a36-b369-446f-af09-46a760c1be05" containerName="main"
Jul 25 02:02:22.594263 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.594173    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2eb3a36-b369-446f-af09-46a760c1be05" containerName="main"
Jul 25 02:02:22.594263 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.594183    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6506b033-6c94-4bbf-8cb5-9dcd205db9a5" containerName="main"
Jul 25 02:02:22.598563 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.598540    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.601142 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.601119    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"kube-root-ca.crt\""
Jul 25 02:02:22.601310 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.601189    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"openshift-service-ca.crt\""
Jul 25 02:02:22.601851 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.601830    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"default-dockercfg-hzcjs\""
Jul 25 02:02:22.602018 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.601997    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-299af2\"/\"rate-test-model-299af2-kserve-self-signed-certs\""
Jul 25 02:02:22.606214 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.606189    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"]
Jul 25 02:02:22.747969 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.747935    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-model-cache\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.747969 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.747981    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-dshm\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.748180 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.748016    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgvc5\" (UniqueName: \"kubernetes.io/projected/88a858bf-8987-4a8b-a41a-a5d5d8204c35-kube-api-access-wgvc5\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.748180 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.748053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-home\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.748180 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.748081    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tmp-dir\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.748180 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.748140    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tls-certs\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.849345 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.849228    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-model-cache\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.849345 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.849282    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-dshm\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.849345 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.849313    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgvc5\" (UniqueName: \"kubernetes.io/projected/88a858bf-8987-4a8b-a41a-a5d5d8204c35-kube-api-access-wgvc5\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.849644 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.849368    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-home\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.849644 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.849393    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tmp-dir\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.849644 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.849433    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tls-certs\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.849824 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.849725    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-model-cache\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.849824 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.849768    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-home\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.849898 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.849855    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tmp-dir\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.851675 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.851657    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-dshm\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.851985 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.851970    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tls-certs\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.856566 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.856544    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgvc5\" (UniqueName: \"kubernetes.io/projected/88a858bf-8987-4a8b-a41a-a5d5d8204c35-kube-api-access-wgvc5\") pod \"rate-test-model-299af2-kserve-7bfcc799c5-xv5cm\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") " pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:22.909590 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:22.909561    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:23.039583 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:23.039555    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"]
Jul 25 02:02:23.042043 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:02:23.042017    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88a858bf_8987_4a8b_a41a_a5d5d8204c35.slice/crio-bfa49277a99cd137f47cd83a2b68e5f676be5b58aa073cf3beeb678b6bfc012b WatchSource:0}: Error finding container bfa49277a99cd137f47cd83a2b68e5f676be5b58aa073cf3beeb678b6bfc012b: Status 404 returned error can't find the container with id bfa49277a99cd137f47cd83a2b68e5f676be5b58aa073cf3beeb678b6bfc012b
Jul 25 02:02:23.227001 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:23.226963    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm" event={"ID":"88a858bf-8987-4a8b-a41a-a5d5d8204c35","Type":"ContainerStarted","Data":"a483f1799b7cb61bbf774305642fb823d017e179bda21933e9fc99f4a95029f2"}
Jul 25 02:02:23.227001 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:23.227013    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm" event={"ID":"88a858bf-8987-4a8b-a41a-a5d5d8204c35","Type":"ContainerStarted","Data":"bfa49277a99cd137f47cd83a2b68e5f676be5b58aa073cf3beeb678b6bfc012b"}
Jul 25 02:02:23.243902 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:23.243858    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm" podStartSLOduration=1.243843239 podStartE2EDuration="1.243843239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:02:23.241495249 +0000 UTC m=+3027.351304532" watchObservedRunningTime="2026-07-25 02:02:23.243843239 +0000 UTC m=+3027.353652523"
Jul 25 02:02:32.909985 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:32.909947    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:32.910446 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:32.910027    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:32.922535 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:32.922503    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:02:33.291358 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:02:33.291264    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:03:10.598412 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.598374    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"]
Jul 25 02:03:10.601862 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.601844    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.604516 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.604491    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"kube-root-ca.crt\""
Jul 25 02:03:10.605386 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.605366    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"openshift-service-ca.crt\""
Jul 25 02:03:10.605533 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.605516    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"default-dockercfg-fg9lv\""
Jul 25 02:03:10.605623 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.605606    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ab0ae6\"/\"rate-test-model-ab0ae6-kserve-self-signed-certs\""
Jul 25 02:03:10.610522 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.610499    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"]
Jul 25 02:03:10.659920 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.659886    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-home\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.660095 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.659975    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tls-certs\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.660095 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.660023    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-model-cache\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.660095 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.660043    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tmp-dir\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.660095 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.660076    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwjg5\" (UniqueName: \"kubernetes.io/projected/a984eca0-d076-48f1-a84d-ffb5a14be6f1-kube-api-access-qwjg5\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.660237 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.660103    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-dshm\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.761386 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.761317    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tls-certs\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.761600 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.761408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-model-cache\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.761600 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.761424    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tmp-dir\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.761600 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.761477    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwjg5\" (UniqueName: \"kubernetes.io/projected/a984eca0-d076-48f1-a84d-ffb5a14be6f1-kube-api-access-qwjg5\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.761600 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.761508    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-dshm\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.761600 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.761541    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-home\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.761918 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.761891    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-model-cache\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.761983 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.761955    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-home\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.762066 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.762039    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tmp-dir\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.763991 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.763964    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-dshm\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.764096 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.763971    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tls-certs\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.768741 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.768718    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwjg5\" (UniqueName: \"kubernetes.io/projected/a984eca0-d076-48f1-a84d-ffb5a14be6f1-kube-api-access-qwjg5\") pod \"rate-test-model-ab0ae6-kserve-7677868d79-8ddg8\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") " pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:10.912119 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:10.912085    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:11.039964 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:11.039937    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"]
Jul 25 02:03:11.041631 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:03:11.041608    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda984eca0_d076_48f1_a84d_ffb5a14be6f1.slice/crio-92f22460b9e3578d4210de7b2deadbae120211cc97a964bf8d26683f08805ff3 WatchSource:0}: Error finding container 92f22460b9e3578d4210de7b2deadbae120211cc97a964bf8d26683f08805ff3: Status 404 returned error can't find the container with id 92f22460b9e3578d4210de7b2deadbae120211cc97a964bf8d26683f08805ff3
Jul 25 02:03:11.494449 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:11.494416    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8" event={"ID":"a984eca0-d076-48f1-a84d-ffb5a14be6f1","Type":"ContainerStarted","Data":"da18c6f6832cd66bd1ba8d27f905919ee8e2a255cae99aa627dbf85363f0cd7a"}
Jul 25 02:03:11.494449 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:11.494455    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8" event={"ID":"a984eca0-d076-48f1-a84d-ffb5a14be6f1","Type":"ContainerStarted","Data":"92f22460b9e3578d4210de7b2deadbae120211cc97a964bf8d26683f08805ff3"}
Jul 25 02:03:11.512821 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:11.512766    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8" podStartSLOduration=1.5127506130000001 podStartE2EDuration="1.512750613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:03:11.509987491 +0000 UTC m=+3075.619796776" watchObservedRunningTime="2026-07-25 02:03:11.512750613 +0000 UTC m=+3075.622559897"
Jul 25 02:03:20.913057 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:20.913018    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:20.913506 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:20.913085    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:20.925578 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:20.925554    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:03:21.556658 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:03:21.556624    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:04:25.801767 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:25.801676    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"]
Jul 25 02:04:26.779583 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:26.779547    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"]
Jul 25 02:04:27.803307 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:27.803272    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7953-139993390749232"]
Jul 25 02:04:27.806700 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:27.806682    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390749232"
Jul 25 02:04:27.808817 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:27.808795    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v9cdn\""
Jul 25 02:04:27.814820 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:27.814798    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390749232"]
Jul 25 02:04:27.836575 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:27.836539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55sf2\" (UniqueName: \"kubernetes.io/projected/4a831406-c19d-46f2-ae3c-2f534d8508c4-kube-api-access-55sf2\") pod \"test-curl-7953-139993390749232\" (UID: \"4a831406-c19d-46f2-ae3c-2f534d8508c4\") " pod="opendatahub/test-curl-7953-139993390749232"
Jul 25 02:04:27.937429 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:27.937404    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55sf2\" (UniqueName: \"kubernetes.io/projected/4a831406-c19d-46f2-ae3c-2f534d8508c4-kube-api-access-55sf2\") pod \"test-curl-7953-139993390749232\" (UID: \"4a831406-c19d-46f2-ae3c-2f534d8508c4\") " pod="opendatahub/test-curl-7953-139993390749232"
Jul 25 02:04:27.944685 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:27.944663    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55sf2\" (UniqueName: \"kubernetes.io/projected/4a831406-c19d-46f2-ae3c-2f534d8508c4-kube-api-access-55sf2\") pod \"test-curl-7953-139993390749232\" (UID: \"4a831406-c19d-46f2-ae3c-2f534d8508c4\") " pod="opendatahub/test-curl-7953-139993390749232"
Jul 25 02:04:28.117133 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:28.117050    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390749232"
Jul 25 02:04:28.243381 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:28.243356    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390749232"]
Jul 25 02:04:28.245288 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:04:28.245263    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a831406_c19d_46f2_ae3c_2f534d8508c4.slice/crio-45abd9860a95be0ed81dea222ce99670f937c11efa33f1ae651e55c4c1ef408a WatchSource:0}: Error finding container 45abd9860a95be0ed81dea222ce99670f937c11efa33f1ae651e55c4c1ef408a: Status 404 returned error can't find the container with id 45abd9860a95be0ed81dea222ce99670f937c11efa33f1ae651e55c4c1ef408a
Jul 25 02:04:28.246554 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:28.246532    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 25 02:04:28.889025 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:28.888989    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390749232" event={"ID":"4a831406-c19d-46f2-ae3c-2f534d8508c4","Type":"ContainerStarted","Data":"45abd9860a95be0ed81dea222ce99670f937c11efa33f1ae651e55c4c1ef408a"}
Jul 25 02:04:29.896666 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:29.896626    2572 generic.go:356] "Generic (PLEG): container finished" podID="4a831406-c19d-46f2-ae3c-2f534d8508c4" containerID="adc09d0f0c97b3a800f4f5c53bf7e5f6b68d517400bde22b2334288edb099e07" exitCode=0
Jul 25 02:04:29.897060 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:29.896725    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390749232" event={"ID":"4a831406-c19d-46f2-ae3c-2f534d8508c4","Type":"ContainerDied","Data":"adc09d0f0c97b3a800f4f5c53bf7e5f6b68d517400bde22b2334288edb099e07"}
Jul 25 02:04:31.028779 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.028757    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390749232"
Jul 25 02:04:31.060493 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.060453    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7953-139993390749232_4a831406-c19d-46f2-ae3c-2f534d8508c4/test-curl-7953-139993390749232/0.log"
Jul 25 02:04:31.071730 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.071707    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55sf2\" (UniqueName: \"kubernetes.io/projected/4a831406-c19d-46f2-ae3c-2f534d8508c4-kube-api-access-55sf2\") pod \"4a831406-c19d-46f2-ae3c-2f534d8508c4\" (UID: \"4a831406-c19d-46f2-ae3c-2f534d8508c4\") "
Jul 25 02:04:31.073870 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.073842    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a831406-c19d-46f2-ae3c-2f534d8508c4-kube-api-access-55sf2" (OuterVolumeSpecName: "kube-api-access-55sf2") pod "4a831406-c19d-46f2-ae3c-2f534d8508c4" (UID: "4a831406-c19d-46f2-ae3c-2f534d8508c4"). InnerVolumeSpecName "kube-api-access-55sf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:04:31.083360 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.083334    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7953-139993390749232"]
Jul 25 02:04:31.085432 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.085409    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7953-139993390749232"]
Jul 25 02:04:31.173092 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.173020    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55sf2\" (UniqueName: \"kubernetes.io/projected/4a831406-c19d-46f2-ae3c-2f534d8508c4-kube-api-access-55sf2\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:31.214288 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.214257    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7953-139993390750768"]
Jul 25 02:04:31.214603 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.214590    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a831406-c19d-46f2-ae3c-2f534d8508c4" containerName="test-curl-7953-139993390749232"
Jul 25 02:04:31.214647 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.214605    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a831406-c19d-46f2-ae3c-2f534d8508c4" containerName="test-curl-7953-139993390749232"
Jul 25 02:04:31.214694 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.214679    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a831406-c19d-46f2-ae3c-2f534d8508c4" containerName="test-curl-7953-139993390749232"
Jul 25 02:04:31.217598 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.217582    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390750768"
Jul 25 02:04:31.224211 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.224187    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390750768"]
Jul 25 02:04:31.274430 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.274403    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pzhp\" (UniqueName: \"kubernetes.io/projected/0a972a3e-95a4-4638-a9c6-025a9ca5cf07-kube-api-access-5pzhp\") pod \"test-curl-7953-139993390750768\" (UID: \"0a972a3e-95a4-4638-a9c6-025a9ca5cf07\") " pod="opendatahub/test-curl-7953-139993390750768"
Jul 25 02:04:31.375346 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.375293    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pzhp\" (UniqueName: \"kubernetes.io/projected/0a972a3e-95a4-4638-a9c6-025a9ca5cf07-kube-api-access-5pzhp\") pod \"test-curl-7953-139993390750768\" (UID: \"0a972a3e-95a4-4638-a9c6-025a9ca5cf07\") " pod="opendatahub/test-curl-7953-139993390750768"
Jul 25 02:04:31.382835 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.382814    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pzhp\" (UniqueName: \"kubernetes.io/projected/0a972a3e-95a4-4638-a9c6-025a9ca5cf07-kube-api-access-5pzhp\") pod \"test-curl-7953-139993390750768\" (UID: \"0a972a3e-95a4-4638-a9c6-025a9ca5cf07\") " pod="opendatahub/test-curl-7953-139993390750768"
Jul 25 02:04:31.528418 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.528352    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390750768"
Jul 25 02:04:31.658529 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.658502    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390750768"]
Jul 25 02:04:31.660350 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:04:31.660310    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a972a3e_95a4_4638_a9c6_025a9ca5cf07.slice/crio-c982b4df77785036f832c4cce8c04739473d8c4453e6c6358327196486ddc145 WatchSource:0}: Error finding container c982b4df77785036f832c4cce8c04739473d8c4453e6c6358327196486ddc145: Status 404 returned error can't find the container with id c982b4df77785036f832c4cce8c04739473d8c4453e6c6358327196486ddc145
Jul 25 02:04:31.907921 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.907895    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390749232"
Jul 25 02:04:31.907921 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.907911    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45abd9860a95be0ed81dea222ce99670f937c11efa33f1ae651e55c4c1ef408a"
Jul 25 02:04:31.909170 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:31.909144    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390750768" event={"ID":"0a972a3e-95a4-4638-a9c6-025a9ca5cf07","Type":"ContainerStarted","Data":"c982b4df77785036f832c4cce8c04739473d8c4453e6c6358327196486ddc145"}
Jul 25 02:04:31.918897 ip-10-0-135-99 kubenswrapper[2572]: E0725 02:04:31.918872    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7953-139993390749232\" is forbidden: User \"system:node:ip-10-0-135-99.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-99.ec2.internal' and this object" podUID="4a831406-c19d-46f2-ae3c-2f534d8508c4" pod="opendatahub/test-curl-7953-139993390749232"
Jul 25 02:04:32.353567 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:32.353487    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a831406-c19d-46f2-ae3c-2f534d8508c4" path="/var/lib/kubelet/pods/4a831406-c19d-46f2-ae3c-2f534d8508c4/volumes"
Jul 25 02:04:32.916227 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:32.916195    2572 generic.go:356] "Generic (PLEG): container finished" podID="0a972a3e-95a4-4638-a9c6-025a9ca5cf07" containerID="5e5f5f9bfcb5940c4d1400378189c1e1586cb869df38b47c0626268f7af4692d" exitCode=0
Jul 25 02:04:32.916454 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:32.916280    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390750768" event={"ID":"0a972a3e-95a4-4638-a9c6-025a9ca5cf07","Type":"ContainerDied","Data":"5e5f5f9bfcb5940c4d1400378189c1e1586cb869df38b47c0626268f7af4692d"}
Jul 25 02:04:34.058812 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.058790    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390750768"
Jul 25 02:04:34.082631 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.082572    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7953-139993390750768_0a972a3e-95a4-4638-a9c6-025a9ca5cf07/test-curl-7953-139993390750768/0.log"
Jul 25 02:04:34.100023 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.099997    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pzhp\" (UniqueName: \"kubernetes.io/projected/0a972a3e-95a4-4638-a9c6-025a9ca5cf07-kube-api-access-5pzhp\") pod \"0a972a3e-95a4-4638-a9c6-025a9ca5cf07\" (UID: \"0a972a3e-95a4-4638-a9c6-025a9ca5cf07\") "
Jul 25 02:04:34.102386 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.102356    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a972a3e-95a4-4638-a9c6-025a9ca5cf07-kube-api-access-5pzhp" (OuterVolumeSpecName: "kube-api-access-5pzhp") pod "0a972a3e-95a4-4638-a9c6-025a9ca5cf07" (UID: "0a972a3e-95a4-4638-a9c6-025a9ca5cf07"). InnerVolumeSpecName "kube-api-access-5pzhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:04:34.104337 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.104298    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7953-139993390750768"]
Jul 25 02:04:34.109178 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.109158    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7953-139993390750768"]
Jul 25 02:04:34.201398 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.201285    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pzhp\" (UniqueName: \"kubernetes.io/projected/0a972a3e-95a4-4638-a9c6-025a9ca5cf07-kube-api-access-5pzhp\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:34.273020 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.272986    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7953-139993390747488"]
Jul 25 02:04:34.273366 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.273353    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a972a3e-95a4-4638-a9c6-025a9ca5cf07" containerName="test-curl-7953-139993390750768"
Jul 25 02:04:34.273415 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.273368    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a972a3e-95a4-4638-a9c6-025a9ca5cf07" containerName="test-curl-7953-139993390750768"
Jul 25 02:04:34.273452 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.273441    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a972a3e-95a4-4638-a9c6-025a9ca5cf07" containerName="test-curl-7953-139993390750768"
Jul 25 02:04:34.276231 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.276214    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:34.284133 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.283565    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390747488"]
Jul 25 02:04:34.302308 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.302278    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm6r2\" (UniqueName: \"kubernetes.io/projected/41af2b86-6be7-43a2-8963-5742a0b1ceea-kube-api-access-pm6r2\") pod \"test-curl-7953-139993390747488\" (UID: \"41af2b86-6be7-43a2-8963-5742a0b1ceea\") " pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:34.352827 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.352796    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a972a3e-95a4-4638-a9c6-025a9ca5cf07" path="/var/lib/kubelet/pods/0a972a3e-95a4-4638-a9c6-025a9ca5cf07/volumes"
Jul 25 02:04:34.403093 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.403063    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm6r2\" (UniqueName: \"kubernetes.io/projected/41af2b86-6be7-43a2-8963-5742a0b1ceea-kube-api-access-pm6r2\") pod \"test-curl-7953-139993390747488\" (UID: \"41af2b86-6be7-43a2-8963-5742a0b1ceea\") " pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:34.410960 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.410934    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm6r2\" (UniqueName: \"kubernetes.io/projected/41af2b86-6be7-43a2-8963-5742a0b1ceea-kube-api-access-pm6r2\") pod \"test-curl-7953-139993390747488\" (UID: \"41af2b86-6be7-43a2-8963-5742a0b1ceea\") " pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:34.587654 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.587570    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:34.923228 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.923201    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390747488"]
Jul 25 02:04:34.924641 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:04:34.924606    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41af2b86_6be7_43a2_8963_5742a0b1ceea.slice/crio-63552c8e936ab19244c04c2bce4f67d477eabaf5d531ca74caed5e62d4ed3635 WatchSource:0}: Error finding container 63552c8e936ab19244c04c2bce4f67d477eabaf5d531ca74caed5e62d4ed3635: Status 404 returned error can't find the container with id 63552c8e936ab19244c04c2bce4f67d477eabaf5d531ca74caed5e62d4ed3635
Jul 25 02:04:34.929863 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.929844    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390750768"
Jul 25 02:04:34.929863 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:34.929857    2572 scope.go:117] "RemoveContainer" containerID="5e5f5f9bfcb5940c4d1400378189c1e1586cb869df38b47c0626268f7af4692d"
Jul 25 02:04:35.937431 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:35.937396    2572 generic.go:356] "Generic (PLEG): container finished" podID="41af2b86-6be7-43a2-8963-5742a0b1ceea" containerID="6d51ce45e5eaf740b8eaefe5162cdab6bbcfd1f5892a9b1aeedecd38192ff6b2" exitCode=0
Jul 25 02:04:35.937815 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:35.937449    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390747488" event={"ID":"41af2b86-6be7-43a2-8963-5742a0b1ceea","Type":"ContainerDied","Data":"6d51ce45e5eaf740b8eaefe5162cdab6bbcfd1f5892a9b1aeedecd38192ff6b2"}
Jul 25 02:04:35.937815 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:35.937479    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390747488" event={"ID":"41af2b86-6be7-43a2-8963-5742a0b1ceea","Type":"ContainerStarted","Data":"63552c8e936ab19244c04c2bce4f67d477eabaf5d531ca74caed5e62d4ed3635"}
Jul 25 02:04:37.074145 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.074121    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:37.091859 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.091823    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7953-139993390747488_41af2b86-6be7-43a2-8963-5742a0b1ceea/test-curl-7953-139993390747488/0.log"
Jul 25 02:04:37.113138 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.113109    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7953-139993390747488"]
Jul 25 02:04:37.118421 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.118394    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7953-139993390747488"]
Jul 25 02:04:37.126428 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.126402    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm6r2\" (UniqueName: \"kubernetes.io/projected/41af2b86-6be7-43a2-8963-5742a0b1ceea-kube-api-access-pm6r2\") pod \"41af2b86-6be7-43a2-8963-5742a0b1ceea\" (UID: \"41af2b86-6be7-43a2-8963-5742a0b1ceea\") "
Jul 25 02:04:37.128474 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.128454    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41af2b86-6be7-43a2-8963-5742a0b1ceea-kube-api-access-pm6r2" (OuterVolumeSpecName: "kube-api-access-pm6r2") pod "41af2b86-6be7-43a2-8963-5742a0b1ceea" (UID: "41af2b86-6be7-43a2-8963-5742a0b1ceea"). InnerVolumeSpecName "kube-api-access-pm6r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:04:37.227385 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.227284    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm6r2\" (UniqueName: \"kubernetes.io/projected/41af2b86-6be7-43a2-8963-5742a0b1ceea-kube-api-access-pm6r2\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:37.242354 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.242294    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7953-139993390747488"]
Jul 25 02:04:37.242731 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.242712    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41af2b86-6be7-43a2-8963-5742a0b1ceea" containerName="test-curl-7953-139993390747488"
Jul 25 02:04:37.242814 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.242733    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41af2b86-6be7-43a2-8963-5742a0b1ceea" containerName="test-curl-7953-139993390747488"
Jul 25 02:04:37.242870 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.242846    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41af2b86-6be7-43a2-8963-5742a0b1ceea" containerName="test-curl-7953-139993390747488"
Jul 25 02:04:37.245906 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.245887    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:37.251625 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.251603    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390747488"]
Jul 25 02:04:37.328613 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.328574    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9wbg\" (UniqueName: \"kubernetes.io/projected/549d8b44-2466-4853-8ac9-b0433af13ba1-kube-api-access-q9wbg\") pod \"test-curl-7953-139993390747488\" (UID: \"549d8b44-2466-4853-8ac9-b0433af13ba1\") " pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:37.429697 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.429659    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9wbg\" (UniqueName: \"kubernetes.io/projected/549d8b44-2466-4853-8ac9-b0433af13ba1-kube-api-access-q9wbg\") pod \"test-curl-7953-139993390747488\" (UID: \"549d8b44-2466-4853-8ac9-b0433af13ba1\") " pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:37.436959 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.436931    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9wbg\" (UniqueName: \"kubernetes.io/projected/549d8b44-2466-4853-8ac9-b0433af13ba1-kube-api-access-q9wbg\") pod \"test-curl-7953-139993390747488\" (UID: \"549d8b44-2466-4853-8ac9-b0433af13ba1\") " pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:37.556560 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.556461    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:37.685211 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.685184    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390747488"]
Jul 25 02:04:37.686953 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:04:37.686926    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod549d8b44_2466_4853_8ac9_b0433af13ba1.slice/crio-f95865717b777da8c49e1b2f5651d7b63f474c8c62fc9b94495bc67716f9e459 WatchSource:0}: Error finding container f95865717b777da8c49e1b2f5651d7b63f474c8c62fc9b94495bc67716f9e459: Status 404 returned error can't find the container with id f95865717b777da8c49e1b2f5651d7b63f474c8c62fc9b94495bc67716f9e459
Jul 25 02:04:37.949805 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.949766    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390747488" event={"ID":"549d8b44-2466-4853-8ac9-b0433af13ba1","Type":"ContainerStarted","Data":"f95865717b777da8c49e1b2f5651d7b63f474c8c62fc9b94495bc67716f9e459"}
Jul 25 02:04:37.950909 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.950891    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63552c8e936ab19244c04c2bce4f67d477eabaf5d531ca74caed5e62d4ed3635"
Jul 25 02:04:37.951034 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.950920    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:37.961966 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:37.961940    2572 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7953-139993390747488" oldPodUID="41af2b86-6be7-43a2-8963-5742a0b1ceea" podUID="549d8b44-2466-4853-8ac9-b0433af13ba1"
Jul 25 02:04:38.353163 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:38.353065    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41af2b86-6be7-43a2-8963-5742a0b1ceea" path="/var/lib/kubelet/pods/41af2b86-6be7-43a2-8963-5742a0b1ceea/volumes"
Jul 25 02:04:38.958119 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:38.958085    2572 generic.go:356] "Generic (PLEG): container finished" podID="549d8b44-2466-4853-8ac9-b0433af13ba1" containerID="87e5ddbfb043294f16f99d97c3d77c34d85bc8e826f4c414a3d83e02111c6511" exitCode=0
Jul 25 02:04:38.958307 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:38.958137    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390747488" event={"ID":"549d8b44-2466-4853-8ac9-b0433af13ba1","Type":"ContainerDied","Data":"87e5ddbfb043294f16f99d97c3d77c34d85bc8e826f4c414a3d83e02111c6511"}
Jul 25 02:04:40.099903 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.099878    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:40.118337 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.118297    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7953-139993390747488_549d8b44-2466-4853-8ac9-b0433af13ba1/test-curl-7953-139993390747488/0.log"
Jul 25 02:04:40.139979 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.139946    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7953-139993390747488"]
Jul 25 02:04:40.142108 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.142087    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7953-139993390747488"]
Jul 25 02:04:40.150584 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.150560    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9wbg\" (UniqueName: \"kubernetes.io/projected/549d8b44-2466-4853-8ac9-b0433af13ba1-kube-api-access-q9wbg\") pod \"549d8b44-2466-4853-8ac9-b0433af13ba1\" (UID: \"549d8b44-2466-4853-8ac9-b0433af13ba1\") "
Jul 25 02:04:40.152587 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.152565    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549d8b44-2466-4853-8ac9-b0433af13ba1-kube-api-access-q9wbg" (OuterVolumeSpecName: "kube-api-access-q9wbg") pod "549d8b44-2466-4853-8ac9-b0433af13ba1" (UID: "549d8b44-2466-4853-8ac9-b0433af13ba1"). InnerVolumeSpecName "kube-api-access-q9wbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:04:40.252132 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.252043    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9wbg\" (UniqueName: \"kubernetes.io/projected/549d8b44-2466-4853-8ac9-b0433af13ba1-kube-api-access-q9wbg\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:40.279717 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.279681    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7953-139993390790384"]
Jul 25 02:04:40.280043 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.280031    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549d8b44-2466-4853-8ac9-b0433af13ba1" containerName="test-curl-7953-139993390747488"
Jul 25 02:04:40.280094 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.280044    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="549d8b44-2466-4853-8ac9-b0433af13ba1" containerName="test-curl-7953-139993390747488"
Jul 25 02:04:40.280132 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.280109    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="549d8b44-2466-4853-8ac9-b0433af13ba1" containerName="test-curl-7953-139993390747488"
Jul 25 02:04:40.282911 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.282894    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:40.290493 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.290462    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390790384"]
Jul 25 02:04:40.352878 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.352840    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nd24\" (UniqueName: \"kubernetes.io/projected/a6d8881c-73f9-4c80-87cc-6d3ac749c90e-kube-api-access-4nd24\") pod \"test-curl-7953-139993390790384\" (UID: \"a6d8881c-73f9-4c80-87cc-6d3ac749c90e\") " pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:40.353691 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.353670    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="549d8b44-2466-4853-8ac9-b0433af13ba1" path="/var/lib/kubelet/pods/549d8b44-2466-4853-8ac9-b0433af13ba1/volumes"
Jul 25 02:04:40.453830 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.453798    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nd24\" (UniqueName: \"kubernetes.io/projected/a6d8881c-73f9-4c80-87cc-6d3ac749c90e-kube-api-access-4nd24\") pod \"test-curl-7953-139993390790384\" (UID: \"a6d8881c-73f9-4c80-87cc-6d3ac749c90e\") " pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:40.461567 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.461543    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nd24\" (UniqueName: \"kubernetes.io/projected/a6d8881c-73f9-4c80-87cc-6d3ac749c90e-kube-api-access-4nd24\") pod \"test-curl-7953-139993390790384\" (UID: \"a6d8881c-73f9-4c80-87cc-6d3ac749c90e\") " pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:40.592777 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.592679    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:40.720513 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.720480    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390790384"]
Jul 25 02:04:40.820410 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.820372    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm" podUID="88a858bf-8987-4a8b-a41a-a5d5d8204c35" containerName="main" containerID="cri-o://a483f1799b7cb61bbf774305642fb823d017e179bda21933e9fc99f4a95029f2" gracePeriod=45
Jul 25 02:04:40.970072 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.970032    2572 generic.go:356] "Generic (PLEG): container finished" podID="88a858bf-8987-4a8b-a41a-a5d5d8204c35" containerID="a483f1799b7cb61bbf774305642fb823d017e179bda21933e9fc99f4a95029f2" exitCode=0
Jul 25 02:04:40.970263 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.970115    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm" event={"ID":"88a858bf-8987-4a8b-a41a-a5d5d8204c35","Type":"ContainerDied","Data":"a483f1799b7cb61bbf774305642fb823d017e179bda21933e9fc99f4a95029f2"}
Jul 25 02:04:40.971380 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.971357    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390790384" event={"ID":"a6d8881c-73f9-4c80-87cc-6d3ac749c90e","Type":"ContainerStarted","Data":"a1d174da84348734d04aa1ba29c3b6190e185aa59987001b05a2776ef786c079"}
Jul 25 02:04:40.972749 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.972733    2572 scope.go:117] "RemoveContainer" containerID="87e5ddbfb043294f16f99d97c3d77c34d85bc8e826f4c414a3d83e02111c6511"
Jul 25 02:04:40.972749 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:40.972744    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390747488"
Jul 25 02:04:41.066193 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.066172    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:04:41.160306 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160278    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tmp-dir\") pod \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") "
Jul 25 02:04:41.160727 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160312    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tls-certs\") pod \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") "
Jul 25 02:04:41.160727 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160387    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-model-cache\") pod \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") "
Jul 25 02:04:41.160727 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160423    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-home\") pod \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") "
Jul 25 02:04:41.160727 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160448    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-dshm\") pod \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") "
Jul 25 02:04:41.160727 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160520    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgvc5\" (UniqueName: \"kubernetes.io/projected/88a858bf-8987-4a8b-a41a-a5d5d8204c35-kube-api-access-wgvc5\") pod \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\" (UID: \"88a858bf-8987-4a8b-a41a-a5d5d8204c35\") "
Jul 25 02:04:41.160727 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160546    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88a858bf-8987-4a8b-a41a-a5d5d8204c35" (UID: "88a858bf-8987-4a8b-a41a-a5d5d8204c35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:04:41.160727 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160630    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-model-cache" (OuterVolumeSpecName: "model-cache") pod "88a858bf-8987-4a8b-a41a-a5d5d8204c35" (UID: "88a858bf-8987-4a8b-a41a-a5d5d8204c35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:04:41.160727 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160675    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-home" (OuterVolumeSpecName: "home") pod "88a858bf-8987-4a8b-a41a-a5d5d8204c35" (UID: "88a858bf-8987-4a8b-a41a-a5d5d8204c35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:04:41.161114 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160805    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:41.161114 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160826    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:41.161114 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.160841    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:41.162450 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.162428    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-dshm" (OuterVolumeSpecName: "dshm") pod "88a858bf-8987-4a8b-a41a-a5d5d8204c35" (UID: "88a858bf-8987-4a8b-a41a-a5d5d8204c35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:04:41.162553 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.162499    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88a858bf-8987-4a8b-a41a-a5d5d8204c35" (UID: "88a858bf-8987-4a8b-a41a-a5d5d8204c35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 02:04:41.162553 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.162526    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a858bf-8987-4a8b-a41a-a5d5d8204c35-kube-api-access-wgvc5" (OuterVolumeSpecName: "kube-api-access-wgvc5") pod "88a858bf-8987-4a8b-a41a-a5d5d8204c35" (UID: "88a858bf-8987-4a8b-a41a-a5d5d8204c35"). InnerVolumeSpecName "kube-api-access-wgvc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:04:41.262156 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.262075    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a858bf-8987-4a8b-a41a-a5d5d8204c35-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:41.262156 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.262101    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a858bf-8987-4a8b-a41a-a5d5d8204c35-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:41.262156 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.262111    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgvc5\" (UniqueName: \"kubernetes.io/projected/88a858bf-8987-4a8b-a41a-a5d5d8204c35-kube-api-access-wgvc5\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:41.798340 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.798275    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8" podUID="a984eca0-d076-48f1-a84d-ffb5a14be6f1" containerName="main" containerID="cri-o://da18c6f6832cd66bd1ba8d27f905919ee8e2a255cae99aa627dbf85363f0cd7a" gracePeriod=45
Jul 25 02:04:41.994210 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.990715    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm" event={"ID":"88a858bf-8987-4a8b-a41a-a5d5d8204c35","Type":"ContainerDied","Data":"bfa49277a99cd137f47cd83a2b68e5f676be5b58aa073cf3beeb678b6bfc012b"}
Jul 25 02:04:41.994210 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.990776    2572 scope.go:117] "RemoveContainer" containerID="a483f1799b7cb61bbf774305642fb823d017e179bda21933e9fc99f4a95029f2"
Jul 25 02:04:41.994210 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.990920    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"
Jul 25 02:04:42.000132 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.999680    2572 generic.go:356] "Generic (PLEG): container finished" podID="a6d8881c-73f9-4c80-87cc-6d3ac749c90e" containerID="3df70a97548dc42d925fabd525ca7728d5f6deda9cfd58bafe7d6c961bf66692" exitCode=0
Jul 25 02:04:42.000132 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:41.999801    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390790384" event={"ID":"a6d8881c-73f9-4c80-87cc-6d3ac749c90e","Type":"ContainerDied","Data":"3df70a97548dc42d925fabd525ca7728d5f6deda9cfd58bafe7d6c961bf66692"}
Jul 25 02:04:42.003441 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.003245    2572 generic.go:356] "Generic (PLEG): container finished" podID="a984eca0-d076-48f1-a84d-ffb5a14be6f1" containerID="da18c6f6832cd66bd1ba8d27f905919ee8e2a255cae99aa627dbf85363f0cd7a" exitCode=0
Jul 25 02:04:42.003441 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.003283    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8" event={"ID":"a984eca0-d076-48f1-a84d-ffb5a14be6f1","Type":"ContainerDied","Data":"da18c6f6832cd66bd1ba8d27f905919ee8e2a255cae99aa627dbf85363f0cd7a"}
Jul 25 02:04:42.032170 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.032142    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"]
Jul 25 02:04:42.035923 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.035897    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-299af2/rate-test-model-299af2-kserve-7bfcc799c5-xv5cm"]
Jul 25 02:04:42.065948 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.065925    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:04:42.169104 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.169060    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tmp-dir\") pod \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") "
Jul 25 02:04:42.169104 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.169110    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-model-cache\") pod \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") "
Jul 25 02:04:42.169629 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.169152    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwjg5\" (UniqueName: \"kubernetes.io/projected/a984eca0-d076-48f1-a84d-ffb5a14be6f1-kube-api-access-qwjg5\") pod \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") "
Jul 25 02:04:42.169629 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.169208    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-home\") pod \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") "
Jul 25 02:04:42.169629 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.169262    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tls-certs\") pod \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") "
Jul 25 02:04:42.169629 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.169290    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-dshm\") pod \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\" (UID: \"a984eca0-d076-48f1-a84d-ffb5a14be6f1\") "
Jul 25 02:04:42.169629 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.169405    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a984eca0-d076-48f1-a84d-ffb5a14be6f1" (UID: "a984eca0-d076-48f1-a84d-ffb5a14be6f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:04:42.169629 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.169422    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "a984eca0-d076-48f1-a84d-ffb5a14be6f1" (UID: "a984eca0-d076-48f1-a84d-ffb5a14be6f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:04:42.169629 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.169597    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-home" (OuterVolumeSpecName: "home") pod "a984eca0-d076-48f1-a84d-ffb5a14be6f1" (UID: "a984eca0-d076-48f1-a84d-ffb5a14be6f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:04:42.169629 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.169605    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:42.169933 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.169646    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:42.171361 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.171339    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-dshm" (OuterVolumeSpecName: "dshm") pod "a984eca0-d076-48f1-a84d-ffb5a14be6f1" (UID: "a984eca0-d076-48f1-a84d-ffb5a14be6f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:04:42.171461 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.171434    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a984eca0-d076-48f1-a84d-ffb5a14be6f1" (UID: "a984eca0-d076-48f1-a84d-ffb5a14be6f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 02:04:42.171571 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.171551    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a984eca0-d076-48f1-a84d-ffb5a14be6f1-kube-api-access-qwjg5" (OuterVolumeSpecName: "kube-api-access-qwjg5") pod "a984eca0-d076-48f1-a84d-ffb5a14be6f1" (UID: "a984eca0-d076-48f1-a84d-ffb5a14be6f1"). InnerVolumeSpecName "kube-api-access-qwjg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:04:42.270470 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.270431    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a984eca0-d076-48f1-a84d-ffb5a14be6f1-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:42.270470 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.270465    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:42.270470 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.270474    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwjg5\" (UniqueName: \"kubernetes.io/projected/a984eca0-d076-48f1-a84d-ffb5a14be6f1-kube-api-access-qwjg5\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:42.270711 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.270484    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a984eca0-d076-48f1-a84d-ffb5a14be6f1-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:42.353061 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:42.352980    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a858bf-8987-4a8b-a41a-a5d5d8204c35" path="/var/lib/kubelet/pods/88a858bf-8987-4a8b-a41a-a5d5d8204c35/volumes"
Jul 25 02:04:43.011049 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.011010    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8" event={"ID":"a984eca0-d076-48f1-a84d-ffb5a14be6f1","Type":"ContainerDied","Data":"92f22460b9e3578d4210de7b2deadbae120211cc97a964bf8d26683f08805ff3"}
Jul 25 02:04:43.011243 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.011068    2572 scope.go:117] "RemoveContainer" containerID="da18c6f6832cd66bd1ba8d27f905919ee8e2a255cae99aa627dbf85363f0cd7a"
Jul 25 02:04:43.011243 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.011022    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"
Jul 25 02:04:43.026203 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.026178    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"]
Jul 25 02:04:43.030207 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.030181    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ab0ae6/rate-test-model-ab0ae6-kserve-7677868d79-8ddg8"]
Jul 25 02:04:43.143745 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.143717    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:43.161953 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.161898    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7953-139993390790384_a6d8881c-73f9-4c80-87cc-6d3ac749c90e/test-curl-7953-139993390790384/0.log"
Jul 25 02:04:43.186468 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.186434    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7953-139993390790384"]
Jul 25 02:04:43.188426 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.188406    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7953-139993390790384"]
Jul 25 02:04:43.279000 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.278908    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nd24\" (UniqueName: \"kubernetes.io/projected/a6d8881c-73f9-4c80-87cc-6d3ac749c90e-kube-api-access-4nd24\") pod \"a6d8881c-73f9-4c80-87cc-6d3ac749c90e\" (UID: \"a6d8881c-73f9-4c80-87cc-6d3ac749c90e\") "
Jul 25 02:04:43.280967 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.280941    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d8881c-73f9-4c80-87cc-6d3ac749c90e-kube-api-access-4nd24" (OuterVolumeSpecName: "kube-api-access-4nd24") pod "a6d8881c-73f9-4c80-87cc-6d3ac749c90e" (UID: "a6d8881c-73f9-4c80-87cc-6d3ac749c90e"). InnerVolumeSpecName "kube-api-access-4nd24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:04:43.318288 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.318253    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7953-139993390790384"]
Jul 25 02:04:43.318621 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.318609    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a858bf-8987-4a8b-a41a-a5d5d8204c35" containerName="main"
Jul 25 02:04:43.318668 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.318623    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a858bf-8987-4a8b-a41a-a5d5d8204c35" containerName="main"
Jul 25 02:04:43.318668 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.318641    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6d8881c-73f9-4c80-87cc-6d3ac749c90e" containerName="test-curl-7953-139993390790384"
Jul 25 02:04:43.318668 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.318647    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6d8881c-73f9-4c80-87cc-6d3ac749c90e" containerName="test-curl-7953-139993390790384"
Jul 25 02:04:43.318668 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.318657    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a984eca0-d076-48f1-a84d-ffb5a14be6f1" containerName="main"
Jul 25 02:04:43.318668 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.318663    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a984eca0-d076-48f1-a84d-ffb5a14be6f1" containerName="main"
Jul 25 02:04:43.318825 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.318723    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a984eca0-d076-48f1-a84d-ffb5a14be6f1" containerName="main"
Jul 25 02:04:43.318825 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.318733    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a858bf-8987-4a8b-a41a-a5d5d8204c35" containerName="main"
Jul 25 02:04:43.318825 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.318741    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6d8881c-73f9-4c80-87cc-6d3ac749c90e" containerName="test-curl-7953-139993390790384"
Jul 25 02:04:43.321763 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.321747    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:43.327510 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.327482    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390790384"]
Jul 25 02:04:43.379883 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.379850    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnms4\" (UniqueName: \"kubernetes.io/projected/39295c7b-480c-4ffc-a6f6-6e84bbdd85c4-kube-api-access-pnms4\") pod \"test-curl-7953-139993390790384\" (UID: \"39295c7b-480c-4ffc-a6f6-6e84bbdd85c4\") " pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:43.380025 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.379957    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nd24\" (UniqueName: \"kubernetes.io/projected/a6d8881c-73f9-4c80-87cc-6d3ac749c90e-kube-api-access-4nd24\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:43.480729 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.480686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnms4\" (UniqueName: \"kubernetes.io/projected/39295c7b-480c-4ffc-a6f6-6e84bbdd85c4-kube-api-access-pnms4\") pod \"test-curl-7953-139993390790384\" (UID: \"39295c7b-480c-4ffc-a6f6-6e84bbdd85c4\") " pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:43.488651 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.488617    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnms4\" (UniqueName: \"kubernetes.io/projected/39295c7b-480c-4ffc-a6f6-6e84bbdd85c4-kube-api-access-pnms4\") pod \"test-curl-7953-139993390790384\" (UID: \"39295c7b-480c-4ffc-a6f6-6e84bbdd85c4\") " pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:43.631575 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.631477    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:43.759045 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:43.759009    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7953-139993390790384"]
Jul 25 02:04:43.762181 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:04:43.762156    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39295c7b_480c_4ffc_a6f6_6e84bbdd85c4.slice/crio-be605af0a208806725db424d12efea6341607ecf2d777e72b258705825cddd20 WatchSource:0}: Error finding container be605af0a208806725db424d12efea6341607ecf2d777e72b258705825cddd20: Status 404 returned error can't find the container with id be605af0a208806725db424d12efea6341607ecf2d777e72b258705825cddd20
Jul 25 02:04:44.017086 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:44.017045    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390790384" event={"ID":"39295c7b-480c-4ffc-a6f6-6e84bbdd85c4","Type":"ContainerStarted","Data":"be605af0a208806725db424d12efea6341607ecf2d777e72b258705825cddd20"}
Jul 25 02:04:44.018247 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:44.018223    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:44.018395 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:44.018235    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1d174da84348734d04aa1ba29c3b6190e185aa59987001b05a2776ef786c079"
Jul 25 02:04:44.029236 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:44.029209    2572 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7953-139993390790384" oldPodUID="a6d8881c-73f9-4c80-87cc-6d3ac749c90e" podUID="39295c7b-480c-4ffc-a6f6-6e84bbdd85c4"
Jul 25 02:04:44.353292 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:44.353206    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6d8881c-73f9-4c80-87cc-6d3ac749c90e" path="/var/lib/kubelet/pods/a6d8881c-73f9-4c80-87cc-6d3ac749c90e/volumes"
Jul 25 02:04:44.353657 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:44.353618    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a984eca0-d076-48f1-a84d-ffb5a14be6f1" path="/var/lib/kubelet/pods/a984eca0-d076-48f1-a84d-ffb5a14be6f1/volumes"
Jul 25 02:04:45.026254 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:45.026220    2572 generic.go:356] "Generic (PLEG): container finished" podID="39295c7b-480c-4ffc-a6f6-6e84bbdd85c4" containerID="825088d4eef384eb8e249cd153ce089fd6adca374cc7571298b7788fa7d2ecbb" exitCode=0
Jul 25 02:04:45.026423 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:45.026266    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7953-139993390790384" event={"ID":"39295c7b-480c-4ffc-a6f6-6e84bbdd85c4","Type":"ContainerDied","Data":"825088d4eef384eb8e249cd153ce089fd6adca374cc7571298b7788fa7d2ecbb"}
Jul 25 02:04:46.160755 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:46.160728    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:46.177919 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:46.177892    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7953-139993390790384_39295c7b-480c-4ffc-a6f6-6e84bbdd85c4/test-curl-7953-139993390790384/0.log"
Jul 25 02:04:46.201741 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:46.201707    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7953-139993390790384"]
Jul 25 02:04:46.203816 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:46.203794    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7953-139993390790384"]
Jul 25 02:04:46.206111 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:46.206097    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnms4\" (UniqueName: \"kubernetes.io/projected/39295c7b-480c-4ffc-a6f6-6e84bbdd85c4-kube-api-access-pnms4\") pod \"39295c7b-480c-4ffc-a6f6-6e84bbdd85c4\" (UID: \"39295c7b-480c-4ffc-a6f6-6e84bbdd85c4\") "
Jul 25 02:04:46.208085 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:46.208068    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39295c7b-480c-4ffc-a6f6-6e84bbdd85c4-kube-api-access-pnms4" (OuterVolumeSpecName: "kube-api-access-pnms4") pod "39295c7b-480c-4ffc-a6f6-6e84bbdd85c4" (UID: "39295c7b-480c-4ffc-a6f6-6e84bbdd85c4"). InnerVolumeSpecName "kube-api-access-pnms4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:04:46.307500 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:46.307409    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnms4\" (UniqueName: \"kubernetes.io/projected/39295c7b-480c-4ffc-a6f6-6e84bbdd85c4-kube-api-access-pnms4\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:04:46.353467 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:46.353433    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39295c7b-480c-4ffc-a6f6-6e84bbdd85c4" path="/var/lib/kubelet/pods/39295c7b-480c-4ffc-a6f6-6e84bbdd85c4/volumes"
Jul 25 02:04:47.020122 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.020085    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"]
Jul 25 02:04:47.020576 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.020551    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39295c7b-480c-4ffc-a6f6-6e84bbdd85c4" containerName="test-curl-7953-139993390790384"
Jul 25 02:04:47.020576 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.020574    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="39295c7b-480c-4ffc-a6f6-6e84bbdd85c4" containerName="test-curl-7953-139993390790384"
Jul 25 02:04:47.020725 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.020662    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="39295c7b-480c-4ffc-a6f6-6e84bbdd85c4" containerName="test-curl-7953-139993390790384"
Jul 25 02:04:47.023466 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.023448    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.025682 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.025665    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e22c40-openshift-default-dockercfg-pghb4\""
Jul 25 02:04:47.035701 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.035677    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"]
Jul 25 02:04:47.040907 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.040889    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7953-139993390790384"
Jul 25 02:04:47.040907 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.040898    2572 scope.go:117] "RemoveContainer" containerID="825088d4eef384eb8e249cd153ce089fd6adca374cc7571298b7788fa7d2ecbb"
Jul 25 02:04:47.116259 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.116225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f5f0b-2352-4dc7-87a2-c9821110869e-istiod-ca-cert\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.116452 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.116268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-podinfo\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.116452 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.116308    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-credential-socket\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.116452 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.116400    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-token\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.116625 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.116469    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-data\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.116625 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.116512    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-socket\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.116625 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.116543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42d46\" (UniqueName: \"kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-kube-api-access-42d46\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.116625 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.116575    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-envoy\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.116625 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.116610    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-certs\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.217222 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217185    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-envoy\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.217688 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217249    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-certs\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.217688 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f5f0b-2352-4dc7-87a2-c9821110869e-istiod-ca-cert\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.217688 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217305    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-podinfo\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.217688 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217383    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-credential-socket\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.217688 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217437    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-token\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.217688 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217541    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-data\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.218026 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217826    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-certs\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.218026 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217830    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-credential-socket\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.218026 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217862    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-socket\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.218026 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217855    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-data\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.218026 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.217938    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42d46\" (UniqueName: \"kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-kube-api-access-42d46\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.218388 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.218240    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-socket\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.218585 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.218563    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f5f0b-2352-4dc7-87a2-c9821110869e-istiod-ca-cert\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.220091 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.220072    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-envoy\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.220193 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.220142    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-podinfo\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.225468 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.225445    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42d46\" (UniqueName: \"kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-kube-api-access-42d46\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.225887 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.225865    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-token\") pod \"e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") " pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.334318 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.334221    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:47.471553 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.471518    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"]
Jul 25 02:04:47.472207 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:04:47.472180    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65f5f0b_2352_4dc7_87a2_c9821110869e.slice/crio-ec15e7aeaa83cdb997f03a4fb09999b89ffabe63189952fb1a01ea81357683d0 WatchSource:0}: Error finding container ec15e7aeaa83cdb997f03a4fb09999b89ffabe63189952fb1a01ea81357683d0: Status 404 returned error can't find the container with id ec15e7aeaa83cdb997f03a4fb09999b89ffabe63189952fb1a01ea81357683d0
Jul 25 02:04:47.474686 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.474652    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 02:04:47.474762 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.474728    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 02:04:47.474802 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:47.474767    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 02:04:48.048848 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:48.048807    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" event={"ID":"b65f5f0b-2352-4dc7-87a2-c9821110869e","Type":"ContainerStarted","Data":"f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd"}
Jul 25 02:04:48.049038 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:48.048860    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" event={"ID":"b65f5f0b-2352-4dc7-87a2-c9821110869e","Type":"ContainerStarted","Data":"ec15e7aeaa83cdb997f03a4fb09999b89ffabe63189952fb1a01ea81357683d0"}
Jul 25 02:04:48.071904 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:48.071854    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" podStartSLOduration=1.07183919 podStartE2EDuration="1.07183919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:04:48.068846948 +0000 UTC m=+3172.178656233" watchObservedRunningTime="2026-07-25 02:04:48.07183919 +0000 UTC m=+3172.181648473"
Jul 25 02:04:48.335220 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:48.335134    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:48.339984 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:48.339957    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:49.054709 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:49.054677    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:49.055760 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:49.055737    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:04:50.220563 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.220528    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"]
Jul 25 02:04:50.224311 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.224287    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:50.226712 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.226688    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e22c40-serving-cert\""
Jul 25 02:04:50.234067 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.234043    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"]
Jul 25 02:04:50.345412 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.345369    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e967be-3baf-4441-9ef0-ebe49588c165-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:50.345628 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.345512    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e967be-3baf-4441-9ef0-ebe49588c165-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:50.345628 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.345556    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qfwc\" (UniqueName: \"kubernetes.io/projected/e3e967be-3baf-4441-9ef0-ebe49588c165-kube-api-access-7qfwc\") pod \"maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:50.446245 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.446202    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e967be-3baf-4441-9ef0-ebe49588c165-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:50.446245 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.446243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qfwc\" (UniqueName: \"kubernetes.io/projected/e3e967be-3baf-4441-9ef0-ebe49588c165-kube-api-access-7qfwc\") pod \"maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:50.446497 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.446283    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e967be-3baf-4441-9ef0-ebe49588c165-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:50.446881 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.446859    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e967be-3baf-4441-9ef0-ebe49588c165-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:50.448754 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.448735    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e967be-3baf-4441-9ef0-ebe49588c165-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:50.454435 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.454403    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qfwc\" (UniqueName: \"kubernetes.io/projected/e3e967be-3baf-4441-9ef0-ebe49588c165-kube-api-access-7qfwc\") pod \"maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:50.537600 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.537514    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:50.669022 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:50.668982    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"]
Jul 25 02:04:50.670283 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:04:50.670256    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e967be_3baf_4441_9ef0_ebe49588c165.slice/crio-19802d898facc374b5fd9c7a6279e790ae131d9fa7d8e67bd8ab33161a32a7e5 WatchSource:0}: Error finding container 19802d898facc374b5fd9c7a6279e790ae131d9fa7d8e67bd8ab33161a32a7e5: Status 404 returned error can't find the container with id 19802d898facc374b5fd9c7a6279e790ae131d9fa7d8e67bd8ab33161a32a7e5
Jul 25 02:04:51.065634 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:51.065595    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j" event={"ID":"e3e967be-3baf-4441-9ef0-ebe49588c165","Type":"ContainerStarted","Data":"19802d898facc374b5fd9c7a6279e790ae131d9fa7d8e67bd8ab33161a32a7e5"}
Jul 25 02:04:52.072159 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:52.072125    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j" event={"ID":"e3e967be-3baf-4441-9ef0-ebe49588c165","Type":"ContainerStarted","Data":"43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea"}
Jul 25 02:04:52.072540 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:52.072256    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:04:52.087755 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:52.087708    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j" podStartSLOduration=1.5509786349999999 podStartE2EDuration="2.087694047s" totalImagesPullingTime="536.715412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:04:52.086441849 +0000 UTC m=+3176.196251135" watchObservedRunningTime="2026-07-25 02:04:52.087694047 +0000 UTC m=+3176.197503331"
Jul 25 02:04:58.083006 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:04:58.082974    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:05:05.096522 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:05.096474    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 25 02:05:05.096923 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:05.096542    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 25 02:05:07.155235 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:07.155202    2572 generic.go:356] "Generic (PLEG): container finished" podID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerID="f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd" exitCode=0
Jul 25 02:05:07.155725 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:07.155275    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" event={"ID":"b65f5f0b-2352-4dc7-87a2-c9821110869e","Type":"ContainerDied","Data":"f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd"}
Jul 25 02:05:07.155725 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:07.155675    2572 scope.go:117] "RemoveContainer" containerID="f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd"
Jul 25 02:05:07.156848 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:07.156814    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 02:05:07.156919 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:07.156892    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 02:05:07.156957 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:07.156925    2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 25 02:05:08.166490 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:08.166456    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" event={"ID":"b65f5f0b-2352-4dc7-87a2-c9821110869e","Type":"ContainerStarted","Data":"b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2"}
Jul 25 02:05:08.335078 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:08.334984    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:05:08.335238 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:08.335205    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Jul 25 02:05:08.335286 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:08.335253    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Jul 25 02:05:09.295127 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.295089    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"]
Jul 25 02:05:09.298351 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.298310    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:09.300567 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.300546    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d66e66-serving-cert\""
Jul 25 02:05:09.313189 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.313164    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"]
Jul 25 02:05:09.335500 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.335469    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Jul 25 02:05:09.335628 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.335521    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Jul 25 02:05:09.417906 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.417863    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d45ed7-300a-4acd-b82b-9fa2157b92ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:09.417906 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.417897    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m25b\" (UniqueName: \"kubernetes.io/projected/76d45ed7-300a-4acd-b82b-9fa2157b92ad-kube-api-access-2m25b\") pod \"maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:09.418271 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.417928    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76d45ed7-300a-4acd-b82b-9fa2157b92ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:09.519211 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.519176    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d45ed7-300a-4acd-b82b-9fa2157b92ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:09.519211 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.519211    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m25b\" (UniqueName: \"kubernetes.io/projected/76d45ed7-300a-4acd-b82b-9fa2157b92ad-kube-api-access-2m25b\") pod \"maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:09.519525 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.519242    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76d45ed7-300a-4acd-b82b-9fa2157b92ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:09.519525 ip-10-0-135-99 kubenswrapper[2572]: E0725 02:05:09.519426    2572 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-serving-cert: secret "maas-api-e2e-ipp-b-d66e66-serving-cert" not found
Jul 25 02:05:09.519525 ip-10-0-135-99 kubenswrapper[2572]: E0725 02:05:09.519505    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76d45ed7-300a-4acd-b82b-9fa2157b92ad-maas-api-tls podName:76d45ed7-300a-4acd-b82b-9fa2157b92ad nodeName:}" failed. No retries permitted until 2026-07-25 02:05:10.0194829 +0000 UTC m=+3194.129292167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76d45ed7-300a-4acd-b82b-9fa2157b92ad-maas-api-tls") pod "maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm" (UID: "76d45ed7-300a-4acd-b82b-9fa2157b92ad") : secret "maas-api-e2e-ipp-b-d66e66-serving-cert" not found
Jul 25 02:05:09.519944 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.519922    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d45ed7-300a-4acd-b82b-9fa2157b92ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:09.527264 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:09.527236    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m25b\" (UniqueName: \"kubernetes.io/projected/76d45ed7-300a-4acd-b82b-9fa2157b92ad-kube-api-access-2m25b\") pod \"maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:10.024764 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:10.024731    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76d45ed7-300a-4acd-b82b-9fa2157b92ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:10.027253 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:10.027223    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76d45ed7-300a-4acd-b82b-9fa2157b92ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:10.208923 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:10.208889    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:10.335363 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:10.335299    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Jul 25 02:05:10.335750 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:10.335403    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Jul 25 02:05:10.388516 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:10.388482    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"]
Jul 25 02:05:10.391302 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:05:10.391271    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d45ed7_300a_4acd_b82b_9fa2157b92ad.slice/crio-5f5e1c394e9151124f49bf2803eff81bd6273fc620df395f3417ee6290e389a0 WatchSource:0}: Error finding container 5f5e1c394e9151124f49bf2803eff81bd6273fc620df395f3417ee6290e389a0: Status 404 returned error can't find the container with id 5f5e1c394e9151124f49bf2803eff81bd6273fc620df395f3417ee6290e389a0
Jul 25 02:05:11.185411 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:11.185374    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm" event={"ID":"76d45ed7-300a-4acd-b82b-9fa2157b92ad","Type":"ContainerStarted","Data":"e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77"}
Jul 25 02:05:11.185585 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:11.185427    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm" event={"ID":"76d45ed7-300a-4acd-b82b-9fa2157b92ad","Type":"ContainerStarted","Data":"5f5e1c394e9151124f49bf2803eff81bd6273fc620df395f3417ee6290e389a0"}
Jul 25 02:05:11.185585 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:11.185470    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:11.199069 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:11.199019    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm" podStartSLOduration=1.8206447319999999 podStartE2EDuration="2.199003278s" totalImagesPullingTime="378.358546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:05:11.198966144 +0000 UTC m=+3195.308775447" watchObservedRunningTime="2026-07-25 02:05:11.199003278 +0000 UTC m=+3195.308812564"
Jul 25 02:05:11.335024 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:11.334991    2572 patch_prober.go:28] interesting pod/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Jul 25 02:05:11.335228 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:11.335052    2572 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Jul 25 02:05:12.338620 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:12.338580    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:05:12.339098 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:12.338984    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:05:12.339546 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:12.339525    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:05:17.196698 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:17.196668    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:05:34.098494 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.098463    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"]
Jul 25 02:05:34.102158 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.102141    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.104615 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.104580    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e22c40\"/\"openshift-service-ca.crt\""
Jul 25 02:05:34.105625 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.105561    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e22c40\"/\"default-dockercfg-d7pss\""
Jul 25 02:05:34.105986 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.105615    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e22c40\"/\"kube-root-ca.crt\""
Jul 25 02:05:34.106262 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.105737    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e22c40\"/\"ipp-route-e22c40-kserve-self-signed-certs\""
Jul 25 02:05:34.109929 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.109907    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"]
Jul 25 02:05:34.243108 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.243072    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-tmp-dir\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.243302 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.243140    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-model-cache\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.243302 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.243196    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpjqm\" (UniqueName: \"kubernetes.io/projected/ca4e1408-be53-4dde-80de-30c294cc1a31-kube-api-access-tpjqm\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.243302 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.243221    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-home\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.243474 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.243285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-dshm\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.243474 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.243396    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4e1408-be53-4dde-80de-30c294cc1a31-tls-certs\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.343939 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.343904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-dshm\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.344134 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.343959    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4e1408-be53-4dde-80de-30c294cc1a31-tls-certs\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.344134 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.344012    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-tmp-dir\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.344134 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.344065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-model-cache\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.344134 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.344100    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpjqm\" (UniqueName: \"kubernetes.io/projected/ca4e1408-be53-4dde-80de-30c294cc1a31-kube-api-access-tpjqm\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.344134 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.344124    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-home\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.344518 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.344491    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-tmp-dir\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.344651 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.344500    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-model-cache\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.344651 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.344567    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-home\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.346349 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.346313    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-dshm\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.346482 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.346464    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4e1408-be53-4dde-80de-30c294cc1a31-tls-certs\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.351444 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.351399    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpjqm\" (UniqueName: \"kubernetes.io/projected/ca4e1408-be53-4dde-80de-30c294cc1a31-kube-api-access-tpjqm\") pod \"ipp-route-e22c40-kserve-754df99bc-2xwfz\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") " pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.414393 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.414361    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:34.547636 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:34.547605    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"]
Jul 25 02:05:34.549161 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:05:34.549134    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca4e1408_be53_4dde_80de_30c294cc1a31.slice/crio-29e35279ce2408c4474f0a9d9c90d0fa82d90554fbbc3dd25ce155c418f6bd08 WatchSource:0}: Error finding container 29e35279ce2408c4474f0a9d9c90d0fa82d90554fbbc3dd25ce155c418f6bd08: Status 404 returned error can't find the container with id 29e35279ce2408c4474f0a9d9c90d0fa82d90554fbbc3dd25ce155c418f6bd08
Jul 25 02:05:35.316897 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:35.316860    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz" event={"ID":"ca4e1408-be53-4dde-80de-30c294cc1a31","Type":"ContainerStarted","Data":"56ba0b85aeebb00ba280b6247cb6156bd25a9b733687480538d19ce28329dd22"}
Jul 25 02:05:35.317287 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:35.316915    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz" event={"ID":"ca4e1408-be53-4dde-80de-30c294cc1a31","Type":"ContainerStarted","Data":"29e35279ce2408c4474f0a9d9c90d0fa82d90554fbbc3dd25ce155c418f6bd08"}
Jul 25 02:05:35.333379 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:35.333315    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz" podStartSLOduration=1.33329853 podStartE2EDuration="1.33329853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:05:35.331601194 +0000 UTC m=+3219.441410490" watchObservedRunningTime="2026-07-25 02:05:35.33329853 +0000 UTC m=+3219.443107814"
Jul 25 02:05:44.414952 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:44.414912    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:44.415499 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:44.414981    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:44.427999 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:44.427976    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:05:45.386316 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:05:45.386280    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:06:32.995401 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:32.995362    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"]
Jul 25 02:06:32.999594 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:32.999571    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:33.001715 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.001688    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-a585ca-serving-cert\""
Jul 25 02:06:33.008359 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.008317    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"]
Jul 25 02:06:33.150798 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.150761    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d39b19-47e5-4886-88ef-531fdfecf79b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:33.150967 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.150829    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4749p\" (UniqueName: \"kubernetes.io/projected/14d39b19-47e5-4886-88ef-531fdfecf79b-kube-api-access-4749p\") pod \"maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:33.150967 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.150870    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d39b19-47e5-4886-88ef-531fdfecf79b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:33.251306 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.251224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4749p\" (UniqueName: \"kubernetes.io/projected/14d39b19-47e5-4886-88ef-531fdfecf79b-kube-api-access-4749p\") pod \"maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:33.251306 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.251266    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d39b19-47e5-4886-88ef-531fdfecf79b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:33.251533 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.251317    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d39b19-47e5-4886-88ef-531fdfecf79b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:33.252046 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.252024    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d39b19-47e5-4886-88ef-531fdfecf79b-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:33.253817 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.253794    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d39b19-47e5-4886-88ef-531fdfecf79b-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:33.258982 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.258957    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4749p\" (UniqueName: \"kubernetes.io/projected/14d39b19-47e5-4886-88ef-531fdfecf79b-kube-api-access-4749p\") pod \"maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:33.312990 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.312960    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:33.471820 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.471790    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"]
Jul 25 02:06:33.474947 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:06:33.474908    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d39b19_47e5_4886_88ef_531fdfecf79b.slice/crio-05c7ff171a363ae6993078511ed1b1ee38ab04335d1aa3b14c726f7bc07b01e0 WatchSource:0}: Error finding container 05c7ff171a363ae6993078511ed1b1ee38ab04335d1aa3b14c726f7bc07b01e0: Status 404 returned error can't find the container with id 05c7ff171a363ae6993078511ed1b1ee38ab04335d1aa3b14c726f7bc07b01e0
Jul 25 02:06:33.632678 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:33.632602    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9" event={"ID":"14d39b19-47e5-4886-88ef-531fdfecf79b","Type":"ContainerStarted","Data":"05c7ff171a363ae6993078511ed1b1ee38ab04335d1aa3b14c726f7bc07b01e0"}
Jul 25 02:06:34.638888 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:34.638856    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9" event={"ID":"14d39b19-47e5-4886-88ef-531fdfecf79b","Type":"ContainerStarted","Data":"4297f5d53c71cba16845ab531cdeb0431ea9834041edb641b72400c220fd3b9d"}
Jul 25 02:06:34.639264 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:34.638929    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:34.653143 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:34.653097    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9" podStartSLOduration=2.044696902 podStartE2EDuration="2.653082114s" totalImagesPullingTime="608.385212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:06:34.652621198 +0000 UTC m=+3278.762430477" watchObservedRunningTime="2026-07-25 02:06:34.653082114 +0000 UTC m=+3278.762891396"
Jul 25 02:06:40.650199 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:40.650167    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:58.615705 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:58.615674    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"]
Jul 25 02:06:58.616157 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:58.615901    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9" podUID="14d39b19-47e5-4886-88ef-531fdfecf79b" containerName="maas-api" containerID="cri-o://4297f5d53c71cba16845ab531cdeb0431ea9834041edb641b72400c220fd3b9d" gracePeriod=30
Jul 25 02:06:58.772800 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:58.772764    2572 generic.go:356] "Generic (PLEG): container finished" podID="14d39b19-47e5-4886-88ef-531fdfecf79b" containerID="4297f5d53c71cba16845ab531cdeb0431ea9834041edb641b72400c220fd3b9d" exitCode=0
Jul 25 02:06:58.772978 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:58.772830    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9" event={"ID":"14d39b19-47e5-4886-88ef-531fdfecf79b","Type":"ContainerDied","Data":"4297f5d53c71cba16845ab531cdeb0431ea9834041edb641b72400c220fd3b9d"}
Jul 25 02:06:58.872887 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:58.872827    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:58.965920 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:58.965886    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d39b19-47e5-4886-88ef-531fdfecf79b-maas-api-tls\") pod \"14d39b19-47e5-4886-88ef-531fdfecf79b\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") "
Jul 25 02:06:58.965920 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:58.965921    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d39b19-47e5-4886-88ef-531fdfecf79b-openshift-service-ca\") pod \"14d39b19-47e5-4886-88ef-531fdfecf79b\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") "
Jul 25 02:06:58.966129 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:58.965943    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4749p\" (UniqueName: \"kubernetes.io/projected/14d39b19-47e5-4886-88ef-531fdfecf79b-kube-api-access-4749p\") pod \"14d39b19-47e5-4886-88ef-531fdfecf79b\" (UID: \"14d39b19-47e5-4886-88ef-531fdfecf79b\") "
Jul 25 02:06:58.966375 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:58.966350    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d39b19-47e5-4886-88ef-531fdfecf79b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14d39b19-47e5-4886-88ef-531fdfecf79b" (UID: "14d39b19-47e5-4886-88ef-531fdfecf79b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 02:06:58.968090 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:58.968049    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14d39b19-47e5-4886-88ef-531fdfecf79b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14d39b19-47e5-4886-88ef-531fdfecf79b" (UID: "14d39b19-47e5-4886-88ef-531fdfecf79b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 02:06:58.968180 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:58.968132    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d39b19-47e5-4886-88ef-531fdfecf79b-kube-api-access-4749p" (OuterVolumeSpecName: "kube-api-access-4749p") pod "14d39b19-47e5-4886-88ef-531fdfecf79b" (UID: "14d39b19-47e5-4886-88ef-531fdfecf79b"). InnerVolumeSpecName "kube-api-access-4749p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:06:59.067116 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:59.067082    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14d39b19-47e5-4886-88ef-531fdfecf79b-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:06:59.067116 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:59.067110    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14d39b19-47e5-4886-88ef-531fdfecf79b-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:06:59.067116 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:59.067119    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4749p\" (UniqueName: \"kubernetes.io/projected/14d39b19-47e5-4886-88ef-531fdfecf79b-kube-api-access-4749p\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:06:59.779802 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:59.779764    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9" event={"ID":"14d39b19-47e5-4886-88ef-531fdfecf79b","Type":"ContainerDied","Data":"05c7ff171a363ae6993078511ed1b1ee38ab04335d1aa3b14c726f7bc07b01e0"}
Jul 25 02:06:59.779802 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:59.779784    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"
Jul 25 02:06:59.780375 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:59.779825    2572 scope.go:117] "RemoveContainer" containerID="4297f5d53c71cba16845ab531cdeb0431ea9834041edb641b72400c220fd3b9d"
Jul 25 02:06:59.804540 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:59.804515    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"]
Jul 25 02:06:59.806920 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:06:59.806899    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-a585ca-8455f6945f-7n6z9"]
Jul 25 02:07:00.360405 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:00.360365    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14d39b19-47e5-4886-88ef-531fdfecf79b" path="/var/lib/kubelet/pods/14d39b19-47e5-4886-88ef-531fdfecf79b/volumes"
Jul 25 02:07:13.395216 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.395172    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"]
Jul 25 02:07:13.395836 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.395730    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14d39b19-47e5-4886-88ef-531fdfecf79b" containerName="maas-api"
Jul 25 02:07:13.395836 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.395752    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="14d39b19-47e5-4886-88ef-531fdfecf79b" containerName="maas-api"
Jul 25 02:07:13.395962 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.395877    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="14d39b19-47e5-4886-88ef-531fdfecf79b" containerName="maas-api"
Jul 25 02:07:13.399054 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.399031    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"
Jul 25 02:07:13.401225 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.401199    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 02:07:13.405651 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.405625    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"]
Jul 25 02:07:13.487183 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.487148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f461832-6968-4120-a0d5-615198e08397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2\" (UID: \"7f461832-6968-4120-a0d5-615198e08397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"
Jul 25 02:07:13.588526 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.588498    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f461832-6968-4120-a0d5-615198e08397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2\" (UID: \"7f461832-6968-4120-a0d5-615198e08397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"
Jul 25 02:07:13.589099 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.589078    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f461832-6968-4120-a0d5-615198e08397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2\" (UID: \"7f461832-6968-4120-a0d5-615198e08397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"
Jul 25 02:07:13.712439 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.712371    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"
Jul 25 02:07:13.843789 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.843761    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"]
Jul 25 02:07:13.845969 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:07:13.845944    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f461832_6968_4120_a0d5_615198e08397.slice/crio-b2c66975ecfdf3ca708a990ba78fe52d473f84c517c6dc743f9cf4fd67c3babb WatchSource:0}: Error finding container b2c66975ecfdf3ca708a990ba78fe52d473f84c517c6dc743f9cf4fd67c3babb: Status 404 returned error can't find the container with id b2c66975ecfdf3ca708a990ba78fe52d473f84c517c6dc743f9cf4fd67c3babb
Jul 25 02:07:13.861617 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:13.861588    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2" event={"ID":"7f461832-6968-4120-a0d5-615198e08397","Type":"ContainerStarted","Data":"b2c66975ecfdf3ca708a990ba78fe52d473f84c517c6dc743f9cf4fd67c3babb"}
Jul 25 02:07:14.868717 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:14.868621    2572 generic.go:356] "Generic (PLEG): container finished" podID="7f461832-6968-4120-a0d5-615198e08397" containerID="e0d811ea27c49141157f9396e69fcc0cdab129df207199529c241ab3c7b3443e" exitCode=0
Jul 25 02:07:14.868717 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:14.868664    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2" event={"ID":"7f461832-6968-4120-a0d5-615198e08397","Type":"ContainerDied","Data":"e0d811ea27c49141157f9396e69fcc0cdab129df207199529c241ab3c7b3443e"}
Jul 25 02:07:16.000003 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:15.999977    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"
Jul 25 02:07:16.113591 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:16.113563    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f461832-6968-4120-a0d5-615198e08397-maas-api-service-ca\") pod \"7f461832-6968-4120-a0d5-615198e08397\" (UID: \"7f461832-6968-4120-a0d5-615198e08397\") "
Jul 25 02:07:16.113958 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:16.113937    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f461832-6968-4120-a0d5-615198e08397-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7f461832-6968-4120-a0d5-615198e08397" (UID: "7f461832-6968-4120-a0d5-615198e08397"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 02:07:16.214390 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:16.214363    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f461832-6968-4120-a0d5-615198e08397-maas-api-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:07:16.880886 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:16.880853    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2" event={"ID":"7f461832-6968-4120-a0d5-615198e08397","Type":"ContainerDied","Data":"b2c66975ecfdf3ca708a990ba78fe52d473f84c517c6dc743f9cf4fd67c3babb"}
Jul 25 02:07:16.881069 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:16.880909    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2c66975ecfdf3ca708a990ba78fe52d473f84c517c6dc743f9cf4fd67c3babb"
Jul 25 02:07:16.881069 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:16.880860    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"
Jul 25 02:07:29.298297 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.298213    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"]
Jul 25 02:07:29.298714 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.298468    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j" podUID="e3e967be-3baf-4441-9ef0-ebe49588c165" containerName="maas-api" containerID="cri-o://43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea" gracePeriod=30
Jul 25 02:07:29.539388 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.539362    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:07:29.629664 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.629570    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qfwc\" (UniqueName: \"kubernetes.io/projected/e3e967be-3baf-4441-9ef0-ebe49588c165-kube-api-access-7qfwc\") pod \"e3e967be-3baf-4441-9ef0-ebe49588c165\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") "
Jul 25 02:07:29.629842 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.629683    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e967be-3baf-4441-9ef0-ebe49588c165-maas-api-tls\") pod \"e3e967be-3baf-4441-9ef0-ebe49588c165\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") "
Jul 25 02:07:29.629842 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.629712    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e967be-3baf-4441-9ef0-ebe49588c165-openshift-service-ca\") pod \"e3e967be-3baf-4441-9ef0-ebe49588c165\" (UID: \"e3e967be-3baf-4441-9ef0-ebe49588c165\") "
Jul 25 02:07:29.630125 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.630091    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e967be-3baf-4441-9ef0-ebe49588c165-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3e967be-3baf-4441-9ef0-ebe49588c165" (UID: "e3e967be-3baf-4441-9ef0-ebe49588c165"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 02:07:29.631720 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.631696    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e967be-3baf-4441-9ef0-ebe49588c165-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3e967be-3baf-4441-9ef0-ebe49588c165" (UID: "e3e967be-3baf-4441-9ef0-ebe49588c165"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 02:07:29.631827 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.631775    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e967be-3baf-4441-9ef0-ebe49588c165-kube-api-access-7qfwc" (OuterVolumeSpecName: "kube-api-access-7qfwc") pod "e3e967be-3baf-4441-9ef0-ebe49588c165" (UID: "e3e967be-3baf-4441-9ef0-ebe49588c165"). InnerVolumeSpecName "kube-api-access-7qfwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:07:29.731407 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.731375    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3e967be-3baf-4441-9ef0-ebe49588c165-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:07:29.731407 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.731405    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3e967be-3baf-4441-9ef0-ebe49588c165-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:07:29.731607 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.731419    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qfwc\" (UniqueName: \"kubernetes.io/projected/e3e967be-3baf-4441-9ef0-ebe49588c165-kube-api-access-7qfwc\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:07:29.952630 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.952596    2572 generic.go:356] "Generic (PLEG): container finished" podID="e3e967be-3baf-4441-9ef0-ebe49588c165" containerID="43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea" exitCode=0
Jul 25 02:07:29.952812 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.952638    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"
Jul 25 02:07:29.952812 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.952673    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j" event={"ID":"e3e967be-3baf-4441-9ef0-ebe49588c165","Type":"ContainerDied","Data":"43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea"}
Jul 25 02:07:29.952812 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.952726    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j" event={"ID":"e3e967be-3baf-4441-9ef0-ebe49588c165","Type":"ContainerDied","Data":"19802d898facc374b5fd9c7a6279e790ae131d9fa7d8e67bd8ab33161a32a7e5"}
Jul 25 02:07:29.952812 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.952751    2572 scope.go:117] "RemoveContainer" containerID="43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea"
Jul 25 02:07:29.961525 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.961509    2572 scope.go:117] "RemoveContainer" containerID="43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea"
Jul 25 02:07:29.961766 ip-10-0-135-99 kubenswrapper[2572]: E0725 02:07:29.961748    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea\": container with ID starting with 43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea not found: ID does not exist" containerID="43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea"
Jul 25 02:07:29.961831 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.961778    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea"} err="failed to get container status \"43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea\": rpc error: code = NotFound desc = could not find container \"43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea\": container with ID starting with 43c8fc7a1016fdd96842561cb0a8e179e05b2a8717fc548792d73eb100154dea not found: ID does not exist"
Jul 25 02:07:29.973080 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.973056    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"]
Jul 25 02:07:29.976100 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:29.976077    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e22c40-6dbb445dff-tvh6j"]
Jul 25 02:07:30.356548 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:30.356453    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e967be-3baf-4441-9ef0-ebe49588c165" path="/var/lib/kubelet/pods/e3e967be-3baf-4441-9ef0-ebe49588c165/volumes"
Jul 25 02:07:33.460495 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:33.460454    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"]
Jul 25 02:07:33.464300 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:33.464272    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e22c40-5z8x2"]
Jul 25 02:07:34.352930 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:34.352895    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f461832-6968-4120-a0d5-615198e08397" path="/var/lib/kubelet/pods/7f461832-6968-4120-a0d5-615198e08397/volumes"
Jul 25 02:07:38.839666 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:38.839631    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"]
Jul 25 02:07:53.859060 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:53.859016    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz" podUID="ca4e1408-be53-4dde-80de-30c294cc1a31" containerName="main" containerID="cri-o://56ba0b85aeebb00ba280b6247cb6156bd25a9b733687480538d19ce28329dd22" gracePeriod=45
Jul 25 02:07:54.085246 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.085217    2572 generic.go:356] "Generic (PLEG): container finished" podID="ca4e1408-be53-4dde-80de-30c294cc1a31" containerID="56ba0b85aeebb00ba280b6247cb6156bd25a9b733687480538d19ce28329dd22" exitCode=0
Jul 25 02:07:54.085424 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.085295    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz" event={"ID":"ca4e1408-be53-4dde-80de-30c294cc1a31","Type":"ContainerDied","Data":"56ba0b85aeebb00ba280b6247cb6156bd25a9b733687480538d19ce28329dd22"}
Jul 25 02:07:54.113816 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.113764    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:07:54.140746 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.140718    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpjqm\" (UniqueName: \"kubernetes.io/projected/ca4e1408-be53-4dde-80de-30c294cc1a31-kube-api-access-tpjqm\") pod \"ca4e1408-be53-4dde-80de-30c294cc1a31\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") "
Jul 25 02:07:54.140862 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.140766    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-model-cache\") pod \"ca4e1408-be53-4dde-80de-30c294cc1a31\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") "
Jul 25 02:07:54.140862 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.140791    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-home\") pod \"ca4e1408-be53-4dde-80de-30c294cc1a31\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") "
Jul 25 02:07:54.140862 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.140829    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-tmp-dir\") pod \"ca4e1408-be53-4dde-80de-30c294cc1a31\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") "
Jul 25 02:07:54.140862 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.140856    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-dshm\") pod \"ca4e1408-be53-4dde-80de-30c294cc1a31\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") "
Jul 25 02:07:54.141069 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.140899    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4e1408-be53-4dde-80de-30c294cc1a31-tls-certs\") pod \"ca4e1408-be53-4dde-80de-30c294cc1a31\" (UID: \"ca4e1408-be53-4dde-80de-30c294cc1a31\") "
Jul 25 02:07:54.141128 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.141061    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca4e1408-be53-4dde-80de-30c294cc1a31" (UID: "ca4e1408-be53-4dde-80de-30c294cc1a31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:07:54.141185 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.141165    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca4e1408-be53-4dde-80de-30c294cc1a31" (UID: "ca4e1408-be53-4dde-80de-30c294cc1a31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:07:54.141264 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.141247    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-model-cache\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:07:54.141435 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.141413    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-home" (OuterVolumeSpecName: "home") pod "ca4e1408-be53-4dde-80de-30c294cc1a31" (UID: "ca4e1408-be53-4dde-80de-30c294cc1a31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:07:54.143191 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.143162    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-dshm" (OuterVolumeSpecName: "dshm") pod "ca4e1408-be53-4dde-80de-30c294cc1a31" (UID: "ca4e1408-be53-4dde-80de-30c294cc1a31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:07:54.143471 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.143451    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4e1408-be53-4dde-80de-30c294cc1a31-kube-api-access-tpjqm" (OuterVolumeSpecName: "kube-api-access-tpjqm") pod "ca4e1408-be53-4dde-80de-30c294cc1a31" (UID: "ca4e1408-be53-4dde-80de-30c294cc1a31"). InnerVolumeSpecName "kube-api-access-tpjqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:07:54.143547 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.143469    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4e1408-be53-4dde-80de-30c294cc1a31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca4e1408-be53-4dde-80de-30c294cc1a31" (UID: "ca4e1408-be53-4dde-80de-30c294cc1a31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 02:07:54.241990 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.241958    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca4e1408-be53-4dde-80de-30c294cc1a31-tls-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:07:54.241990 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.241986    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpjqm\" (UniqueName: \"kubernetes.io/projected/ca4e1408-be53-4dde-80de-30c294cc1a31-kube-api-access-tpjqm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:07:54.242162 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.241995    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-home\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:07:54.242162 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.242005    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-tmp-dir\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:07:54.242162 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:54.242012    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca4e1408-be53-4dde-80de-30c294cc1a31-dshm\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:07:55.091374 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:55.091318    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"
Jul 25 02:07:55.091374 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:55.091362    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz" event={"ID":"ca4e1408-be53-4dde-80de-30c294cc1a31","Type":"ContainerDied","Data":"29e35279ce2408c4474f0a9d9c90d0fa82d90554fbbc3dd25ce155c418f6bd08"}
Jul 25 02:07:55.091885 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:55.091422    2572 scope.go:117] "RemoveContainer" containerID="56ba0b85aeebb00ba280b6247cb6156bd25a9b733687480538d19ce28329dd22"
Jul 25 02:07:55.106473 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:55.106443    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"]
Jul 25 02:07:55.110719 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:55.110696    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e22c40/ipp-route-e22c40-kserve-754df99bc-2xwfz"]
Jul 25 02:07:56.361425 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:07:56.361392    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca4e1408-be53-4dde-80de-30c294cc1a31" path="/var/lib/kubelet/pods/ca4e1408-be53-4dde-80de-30c294cc1a31/volumes"
Jul 25 02:08:17.737438 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:17.737397    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"]
Jul 25 02:08:17.737944 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:17.737647    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy" containerID="cri-o://b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2" gracePeriod=30
Jul 25 02:08:18.129717 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.129637    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"]
Jul 25 02:08:18.129979 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.129967    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4e1408-be53-4dde-80de-30c294cc1a31" containerName="main"
Jul 25 02:08:18.130035 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.129981    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4e1408-be53-4dde-80de-30c294cc1a31" containerName="main"
Jul 25 02:08:18.130035 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.129997    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e967be-3baf-4441-9ef0-ebe49588c165" containerName="maas-api"
Jul 25 02:08:18.130035 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.130003    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e967be-3baf-4441-9ef0-ebe49588c165" containerName="maas-api"
Jul 25 02:08:18.130035 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.130018    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f461832-6968-4120-a0d5-615198e08397" containerName="revoke-keys"
Jul 25 02:08:18.130035 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.130025    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f461832-6968-4120-a0d5-615198e08397" containerName="revoke-keys"
Jul 25 02:08:18.130210 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.130087    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4e1408-be53-4dde-80de-30c294cc1a31" containerName="main"
Jul 25 02:08:18.130210 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.130095    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e967be-3baf-4441-9ef0-ebe49588c165" containerName="maas-api"
Jul 25 02:08:18.130210 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.130102    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f461832-6968-4120-a0d5-615198e08397" containerName="revoke-keys"
Jul 25 02:08:18.133118 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.133101    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"
Jul 25 02:08:18.135149 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.135119    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 02:08:18.139699 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.139612    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"]
Jul 25 02:08:18.258708 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.258673    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fa05de-c2ab-48ee-b070-b30fa657335e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf\" (UID: \"61fa05de-c2ab-48ee-b070-b30fa657335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"
Jul 25 02:08:18.359658 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.359625    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fa05de-c2ab-48ee-b070-b30fa657335e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf\" (UID: \"61fa05de-c2ab-48ee-b070-b30fa657335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"
Jul 25 02:08:18.360256 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.360236    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fa05de-c2ab-48ee-b070-b30fa657335e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf\" (UID: \"61fa05de-c2ab-48ee-b070-b30fa657335e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"
Jul 25 02:08:18.445293 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.445266    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"
Jul 25 02:08:18.573974 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:18.573949    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"]
Jul 25 02:08:18.575873 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:08:18.575843    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fa05de_c2ab_48ee_b070_b30fa657335e.slice/crio-d8f403126c4f317b2ab182fd97ea5df269ef7600487261c49a3fea41e4ab0d88 WatchSource:0}: Error finding container d8f403126c4f317b2ab182fd97ea5df269ef7600487261c49a3fea41e4ab0d88: Status 404 returned error can't find the container with id d8f403126c4f317b2ab182fd97ea5df269ef7600487261c49a3fea41e4ab0d88
Jul 25 02:08:19.225186 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:19.225137    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf" event={"ID":"61fa05de-c2ab-48ee-b070-b30fa657335e","Type":"ContainerStarted","Data":"4beabf0a3540e8b6667bd7763ed7fa1b616a13edf2b1a0ca6d6cf5861e90dbfa"}
Jul 25 02:08:19.225575 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:19.225217    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf" event={"ID":"61fa05de-c2ab-48ee-b070-b30fa657335e","Type":"ContainerStarted","Data":"d8f403126c4f317b2ab182fd97ea5df269ef7600487261c49a3fea41e4ab0d88"}
Jul 25 02:08:19.239104 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:19.239055    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf" podStartSLOduration=1.239038479 podStartE2EDuration="1.239038479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:08:19.23795538 +0000 UTC m=+3383.347764665" watchObservedRunningTime="2026-07-25 02:08:19.239038479 +0000 UTC m=+3383.348847763"
Jul 25 02:08:20.231404 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:20.231370    2572 generic.go:356] "Generic (PLEG): container finished" podID="61fa05de-c2ab-48ee-b070-b30fa657335e" containerID="4beabf0a3540e8b6667bd7763ed7fa1b616a13edf2b1a0ca6d6cf5861e90dbfa" exitCode=0
Jul 25 02:08:20.231828 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:20.231451    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf" event={"ID":"61fa05de-c2ab-48ee-b070-b30fa657335e","Type":"ContainerDied","Data":"4beabf0a3540e8b6667bd7763ed7fa1b616a13edf2b1a0ca6d6cf5861e90dbfa"}
Jul 25 02:08:21.366931 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:21.366907    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"
Jul 25 02:08:21.485951 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:21.485911    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fa05de-c2ab-48ee-b070-b30fa657335e-maas-api-service-ca\") pod \"61fa05de-c2ab-48ee-b070-b30fa657335e\" (UID: \"61fa05de-c2ab-48ee-b070-b30fa657335e\") "
Jul 25 02:08:21.486244 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:21.486224    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fa05de-c2ab-48ee-b070-b30fa657335e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61fa05de-c2ab-48ee-b070-b30fa657335e" (UID: "61fa05de-c2ab-48ee-b070-b30fa657335e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 02:08:21.587220 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:21.587145    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fa05de-c2ab-48ee-b070-b30fa657335e-maas-api-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:22.243397 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:22.243363    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"
Jul 25 02:08:22.243397 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:22.243373    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf" event={"ID":"61fa05de-c2ab-48ee-b070-b30fa657335e","Type":"ContainerDied","Data":"d8f403126c4f317b2ab182fd97ea5df269ef7600487261c49a3fea41e4ab0d88"}
Jul 25 02:08:22.243617 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:22.243430    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f403126c4f317b2ab182fd97ea5df269ef7600487261c49a3fea41e4ab0d88"
Jul 25 02:08:23.001781 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.001709    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:08:23.099523 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.099494    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-envoy\") pod \"b65f5f0b-2352-4dc7-87a2-c9821110869e\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") "
Jul 25 02:08:23.099716 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.099533    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f5f0b-2352-4dc7-87a2-c9821110869e-istiod-ca-cert\") pod \"b65f5f0b-2352-4dc7-87a2-c9821110869e\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") "
Jul 25 02:08:23.099716 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.099560    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-token\") pod \"b65f5f0b-2352-4dc7-87a2-c9821110869e\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") "
Jul 25 02:08:23.099716 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.099597    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-data\") pod \"b65f5f0b-2352-4dc7-87a2-c9821110869e\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") "
Jul 25 02:08:23.099716 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.099615    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-credential-socket\") pod \"b65f5f0b-2352-4dc7-87a2-c9821110869e\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") "
Jul 25 02:08:23.099716 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.099636    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-certs\") pod \"b65f5f0b-2352-4dc7-87a2-c9821110869e\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") "
Jul 25 02:08:23.099716 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.099679    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-podinfo\") pod \"b65f5f0b-2352-4dc7-87a2-c9821110869e\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") "
Jul 25 02:08:23.099716 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.099720    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-socket\") pod \"b65f5f0b-2352-4dc7-87a2-c9821110869e\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") "
Jul 25 02:08:23.100131 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.099759    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42d46\" (UniqueName: \"kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-kube-api-access-42d46\") pod \"b65f5f0b-2352-4dc7-87a2-c9821110869e\" (UID: \"b65f5f0b-2352-4dc7-87a2-c9821110869e\") "
Jul 25 02:08:23.100131 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.099929    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b65f5f0b-2352-4dc7-87a2-c9821110869e" (UID: "b65f5f0b-2352-4dc7-87a2-c9821110869e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:08:23.100131 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.099947    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65f5f0b-2352-4dc7-87a2-c9821110869e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b65f5f0b-2352-4dc7-87a2-c9821110869e" (UID: "b65f5f0b-2352-4dc7-87a2-c9821110869e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 02:08:23.100295 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.100190    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f5f0b-2352-4dc7-87a2-c9821110869e-istiod-ca-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:23.100295 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.100243    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-credential-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:23.100295 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.100189    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b65f5f0b-2352-4dc7-87a2-c9821110869e" (UID: "b65f5f0b-2352-4dc7-87a2-c9821110869e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:08:23.100295 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.100217    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b65f5f0b-2352-4dc7-87a2-c9821110869e" (UID: "b65f5f0b-2352-4dc7-87a2-c9821110869e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:08:23.100781 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.100754    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-data" (OuterVolumeSpecName: "istio-data") pod "b65f5f0b-2352-4dc7-87a2-c9821110869e" (UID: "b65f5f0b-2352-4dc7-87a2-c9821110869e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:08:23.101768 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.101745    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b65f5f0b-2352-4dc7-87a2-c9821110869e" (UID: "b65f5f0b-2352-4dc7-87a2-c9821110869e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:08:23.101975 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.101954    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-kube-api-access-42d46" (OuterVolumeSpecName: "kube-api-access-42d46") pod "b65f5f0b-2352-4dc7-87a2-c9821110869e" (UID: "b65f5f0b-2352-4dc7-87a2-c9821110869e"). InnerVolumeSpecName "kube-api-access-42d46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:08:23.102037 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.101987    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b65f5f0b-2352-4dc7-87a2-c9821110869e" (UID: "b65f5f0b-2352-4dc7-87a2-c9821110869e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 25 02:08:23.102037 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.102018    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-token" (OuterVolumeSpecName: "istio-token") pod "b65f5f0b-2352-4dc7-87a2-c9821110869e" (UID: "b65f5f0b-2352-4dc7-87a2-c9821110869e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:08:23.200863 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.200837    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42d46\" (UniqueName: \"kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-kube-api-access-42d46\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:23.200863 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.200858    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-envoy\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:23.200863 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.200867    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-token\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:23.201043 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.200876    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-data\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:23.201043 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.200884    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:23.201043 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.200891    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f5f0b-2352-4dc7-87a2-c9821110869e-istio-podinfo\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:23.201043 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.200899    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f5f0b-2352-4dc7-87a2-c9821110869e-workload-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:23.250701 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.250676    2572 generic.go:356] "Generic (PLEG): container finished" podID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerID="b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2" exitCode=0
Jul 25 02:08:23.250819 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.250724    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"
Jul 25 02:08:23.250819 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.250749    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" event={"ID":"b65f5f0b-2352-4dc7-87a2-c9821110869e","Type":"ContainerDied","Data":"b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2"}
Jul 25 02:08:23.250819 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.250790    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w" event={"ID":"b65f5f0b-2352-4dc7-87a2-c9821110869e","Type":"ContainerDied","Data":"ec15e7aeaa83cdb997f03a4fb09999b89ffabe63189952fb1a01ea81357683d0"}
Jul 25 02:08:23.250819 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.250811    2572 scope.go:117] "RemoveContainer" containerID="b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2"
Jul 25 02:08:23.260538 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.260522    2572 scope.go:117] "RemoveContainer" containerID="f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd"
Jul 25 02:08:23.272750 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.272726    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"]
Jul 25 02:08:23.275420 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.275401    2572 scope.go:117] "RemoveContainer" containerID="b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2"
Jul 25 02:08:23.275615 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.275597    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e22c40-openshift-default-5cb96bb486-jr79w"]
Jul 25 02:08:23.275710 ip-10-0-135-99 kubenswrapper[2572]: E0725 02:08:23.275689    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2\": container with ID starting with b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2 not found: ID does not exist" containerID="b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2"
Jul 25 02:08:23.275777 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.275722    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2"} err="failed to get container status \"b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2\": rpc error: code = NotFound desc = could not find container \"b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2\": container with ID starting with b95ae76e29b38a0d0f729163f05f37201146ae48d8686ff1a951a94dcc27ddb2 not found: ID does not exist"
Jul 25 02:08:23.275777 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.275747    2572 scope.go:117] "RemoveContainer" containerID="f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd"
Jul 25 02:08:23.275981 ip-10-0-135-99 kubenswrapper[2572]: E0725 02:08:23.275964    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd\": container with ID starting with f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd not found: ID does not exist" containerID="f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd"
Jul 25 02:08:23.276022 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:23.275987    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd"} err="failed to get container status \"f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd\": rpc error: code = NotFound desc = could not find container \"f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd\": container with ID starting with f019d897c2ab171cbe39382c149a81eda36023cd41411c9fe69f92d608b4f9cd not found: ID does not exist"
Jul 25 02:08:24.352972 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:24.352942    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" path="/var/lib/kubelet/pods/b65f5f0b-2352-4dc7-87a2-c9821110869e/volumes"
Jul 25 02:08:28.483901 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.483860    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"]
Jul 25 02:08:28.485306 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.485009    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm" podUID="76d45ed7-300a-4acd-b82b-9fa2157b92ad" containerName="maas-api" containerID="cri-o://e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77" gracePeriod=30
Jul 25 02:08:28.729715 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.729688    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:08:28.849729 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.849653    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m25b\" (UniqueName: \"kubernetes.io/projected/76d45ed7-300a-4acd-b82b-9fa2157b92ad-kube-api-access-2m25b\") pod \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") "
Jul 25 02:08:28.849729 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.849709    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d45ed7-300a-4acd-b82b-9fa2157b92ad-openshift-service-ca\") pod \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") "
Jul 25 02:08:28.849729 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.849729    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76d45ed7-300a-4acd-b82b-9fa2157b92ad-maas-api-tls\") pod \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\" (UID: \"76d45ed7-300a-4acd-b82b-9fa2157b92ad\") "
Jul 25 02:08:28.850139 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.850111    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d45ed7-300a-4acd-b82b-9fa2157b92ad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76d45ed7-300a-4acd-b82b-9fa2157b92ad" (UID: "76d45ed7-300a-4acd-b82b-9fa2157b92ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 02:08:28.851754 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.851732    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d45ed7-300a-4acd-b82b-9fa2157b92ad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76d45ed7-300a-4acd-b82b-9fa2157b92ad" (UID: "76d45ed7-300a-4acd-b82b-9fa2157b92ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 02:08:28.851869 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.851835    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d45ed7-300a-4acd-b82b-9fa2157b92ad-kube-api-access-2m25b" (OuterVolumeSpecName: "kube-api-access-2m25b") pod "76d45ed7-300a-4acd-b82b-9fa2157b92ad" (UID: "76d45ed7-300a-4acd-b82b-9fa2157b92ad"). InnerVolumeSpecName "kube-api-access-2m25b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:08:28.951109 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.951075    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m25b\" (UniqueName: \"kubernetes.io/projected/76d45ed7-300a-4acd-b82b-9fa2157b92ad-kube-api-access-2m25b\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:28.951109 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.951103    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d45ed7-300a-4acd-b82b-9fa2157b92ad-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:28.951109 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:28.951113    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76d45ed7-300a-4acd-b82b-9fa2157b92ad-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:29.284743 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:29.284710    2572 generic.go:356] "Generic (PLEG): container finished" podID="76d45ed7-300a-4acd-b82b-9fa2157b92ad" containerID="e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77" exitCode=0
Jul 25 02:08:29.284743 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:29.284747    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"
Jul 25 02:08:29.284952 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:29.284751    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm" event={"ID":"76d45ed7-300a-4acd-b82b-9fa2157b92ad","Type":"ContainerDied","Data":"e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77"}
Jul 25 02:08:29.284952 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:29.284883    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm" event={"ID":"76d45ed7-300a-4acd-b82b-9fa2157b92ad","Type":"ContainerDied","Data":"5f5e1c394e9151124f49bf2803eff81bd6273fc620df395f3417ee6290e389a0"}
Jul 25 02:08:29.284952 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:29.284911    2572 scope.go:117] "RemoveContainer" containerID="e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77"
Jul 25 02:08:29.294605 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:29.294585    2572 scope.go:117] "RemoveContainer" containerID="e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77"
Jul 25 02:08:29.294839 ip-10-0-135-99 kubenswrapper[2572]: E0725 02:08:29.294820    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77\": container with ID starting with e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77 not found: ID does not exist" containerID="e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77"
Jul 25 02:08:29.294912 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:29.294846    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77"} err="failed to get container status \"e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77\": rpc error: code = NotFound desc = could not find container \"e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77\": container with ID starting with e4767314c4e05735bd0c199bc41331fc31c4e32ce14b2150099f5b221ac68b77 not found: ID does not exist"
Jul 25 02:08:29.305351 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:29.305308    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"]
Jul 25 02:08:29.308745 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:29.308724    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d66e66-757bdc8c47-vgpkm"]
Jul 25 02:08:30.353128 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:30.353092    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76d45ed7-300a-4acd-b82b-9fa2157b92ad" path="/var/lib/kubelet/pods/76d45ed7-300a-4acd-b82b-9fa2157b92ad/volumes"
Jul 25 02:08:33.194498 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:33.194462    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"]
Jul 25 02:08:33.196759 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:33.196729    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d66e66-d2zjf"]
Jul 25 02:08:34.352811 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:34.352774    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61fa05de-c2ab-48ee-b070-b30fa657335e" path="/var/lib/kubelet/pods/61fa05de-c2ab-48ee-b070-b30fa657335e/volumes"
Jul 25 02:08:39.819023 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.818981    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"]
Jul 25 02:08:39.819521 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819482    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d45ed7-300a-4acd-b82b-9fa2157b92ad" containerName="maas-api"
Jul 25 02:08:39.819521 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819502    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d45ed7-300a-4acd-b82b-9fa2157b92ad" containerName="maas-api"
Jul 25 02:08:39.819521 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819516    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy"
Jul 25 02:08:39.819696 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819526    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy"
Jul 25 02:08:39.819696 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819547    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy"
Jul 25 02:08:39.819696 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819556    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy"
Jul 25 02:08:39.819696 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819580    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61fa05de-c2ab-48ee-b070-b30fa657335e" containerName="revoke-keys"
Jul 25 02:08:39.819696 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819588    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="61fa05de-c2ab-48ee-b070-b30fa657335e" containerName="revoke-keys"
Jul 25 02:08:39.819696 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819674    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="61fa05de-c2ab-48ee-b070-b30fa657335e" containerName="revoke-keys"
Jul 25 02:08:39.819696 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819686    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy"
Jul 25 02:08:39.819696 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819698    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b65f5f0b-2352-4dc7-87a2-c9821110869e" containerName="istio-proxy"
Jul 25 02:08:39.820085 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.819707    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="76d45ed7-300a-4acd-b82b-9fa2157b92ad" containerName="maas-api"
Jul 25 02:08:39.822733 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.822709    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"
Jul 25 02:08:39.824891 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.824862    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 02:08:39.828487 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.828462    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"]
Jul 25 02:08:39.840549 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.840526    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de44c725-7082-4f3e-8d4c-a4f18a261fc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-299af2-hshkj\" (UID: \"de44c725-7082-4f3e-8d4c-a4f18a261fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"
Jul 25 02:08:39.941513 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.941486    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de44c725-7082-4f3e-8d4c-a4f18a261fc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-299af2-hshkj\" (UID: \"de44c725-7082-4f3e-8d4c-a4f18a261fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"
Jul 25 02:08:39.942069 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:39.942050    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de44c725-7082-4f3e-8d4c-a4f18a261fc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-299af2-hshkj\" (UID: \"de44c725-7082-4f3e-8d4c-a4f18a261fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"
Jul 25 02:08:40.134882 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:40.134802    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"
Jul 25 02:08:40.262736 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:40.262708    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"]
Jul 25 02:08:40.352488 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:40.352459    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj" event={"ID":"de44c725-7082-4f3e-8d4c-a4f18a261fc2","Type":"ContainerStarted","Data":"a959cb791d5af8d6c6a724cdc6af09dcae19f8212937841f61b5ac5ecd5e267a"}
Jul 25 02:08:41.354093 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:41.354002    2572 generic.go:356] "Generic (PLEG): container finished" podID="de44c725-7082-4f3e-8d4c-a4f18a261fc2" containerID="6e110de42a0817641a09ae6433d539669971f1b1c0a08c91156c6d2b1077901d" exitCode=0
Jul 25 02:08:41.354093 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:41.354048    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj" event={"ID":"de44c725-7082-4f3e-8d4c-a4f18a261fc2","Type":"ContainerDied","Data":"6e110de42a0817641a09ae6433d539669971f1b1c0a08c91156c6d2b1077901d"}
Jul 25 02:08:42.482369 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:42.482346    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"
Jul 25 02:08:42.563790 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:42.563753    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de44c725-7082-4f3e-8d4c-a4f18a261fc2-maas-api-service-ca\") pod \"de44c725-7082-4f3e-8d4c-a4f18a261fc2\" (UID: \"de44c725-7082-4f3e-8d4c-a4f18a261fc2\") "
Jul 25 02:08:42.564119 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:42.564098    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de44c725-7082-4f3e-8d4c-a4f18a261fc2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de44c725-7082-4f3e-8d4c-a4f18a261fc2" (UID: "de44c725-7082-4f3e-8d4c-a4f18a261fc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 02:08:42.665093 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:42.665066    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de44c725-7082-4f3e-8d4c-a4f18a261fc2-maas-api-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:43.365505 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:43.365473    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"
Jul 25 02:08:43.365682 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:43.365514    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj" event={"ID":"de44c725-7082-4f3e-8d4c-a4f18a261fc2","Type":"ContainerDied","Data":"a959cb791d5af8d6c6a724cdc6af09dcae19f8212937841f61b5ac5ecd5e267a"}
Jul 25 02:08:43.365682 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:43.365559    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a959cb791d5af8d6c6a724cdc6af09dcae19f8212937841f61b5ac5ecd5e267a"
Jul 25 02:08:50.170153 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.170062    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"]
Jul 25 02:08:50.170618 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.170297    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf" podUID="e1116a67-a2e0-4537-9868-e450307e7d32" containerName="maas-api" containerID="cri-o://4aafe11c009886cff0d2fb2137126767e31d4b55300b884df65267111398285b" gracePeriod=30
Jul 25 02:08:50.194790 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.194763    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth"]
Jul 25 02:08:50.197024 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.197001    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-299af2-29749080-2dsth"]
Jul 25 02:08:50.354205 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.354171    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57da19f1-6349-4ef4-b6fc-072011bccbba" path="/var/lib/kubelet/pods/57da19f1-6349-4ef4-b6fc-072011bccbba/volumes"
Jul 25 02:08:50.403564 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.403536    2572 generic.go:356] "Generic (PLEG): container finished" podID="e1116a67-a2e0-4537-9868-e450307e7d32" containerID="4aafe11c009886cff0d2fb2137126767e31d4b55300b884df65267111398285b" exitCode=0
Jul 25 02:08:50.403683 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.403603    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf" event={"ID":"e1116a67-a2e0-4537-9868-e450307e7d32","Type":"ContainerDied","Data":"4aafe11c009886cff0d2fb2137126767e31d4b55300b884df65267111398285b"}
Jul 25 02:08:50.416796 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.416770    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 02:08:50.528631 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.528546    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1116a67-a2e0-4537-9868-e450307e7d32-maas-api-tls\") pod \"e1116a67-a2e0-4537-9868-e450307e7d32\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") "
Jul 25 02:08:50.528631 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.528607    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jbpq\" (UniqueName: \"kubernetes.io/projected/e1116a67-a2e0-4537-9868-e450307e7d32-kube-api-access-7jbpq\") pod \"e1116a67-a2e0-4537-9868-e450307e7d32\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") "
Jul 25 02:08:50.528820 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.528645    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1116a67-a2e0-4537-9868-e450307e7d32-openshift-service-ca\") pod \"e1116a67-a2e0-4537-9868-e450307e7d32\" (UID: \"e1116a67-a2e0-4537-9868-e450307e7d32\") "
Jul 25 02:08:50.529070 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.529045    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1116a67-a2e0-4537-9868-e450307e7d32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1116a67-a2e0-4537-9868-e450307e7d32" (UID: "e1116a67-a2e0-4537-9868-e450307e7d32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 02:08:50.530780 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.530750    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1116a67-a2e0-4537-9868-e450307e7d32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1116a67-a2e0-4537-9868-e450307e7d32" (UID: "e1116a67-a2e0-4537-9868-e450307e7d32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 02:08:50.530884 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.530800    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1116a67-a2e0-4537-9868-e450307e7d32-kube-api-access-7jbpq" (OuterVolumeSpecName: "kube-api-access-7jbpq") pod "e1116a67-a2e0-4537-9868-e450307e7d32" (UID: "e1116a67-a2e0-4537-9868-e450307e7d32"). InnerVolumeSpecName "kube-api-access-7jbpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:08:50.629613 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.629579    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1116a67-a2e0-4537-9868-e450307e7d32-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:50.629613 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.629608    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1116a67-a2e0-4537-9868-e450307e7d32-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:50.629613 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:50.629618    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jbpq\" (UniqueName: \"kubernetes.io/projected/e1116a67-a2e0-4537-9868-e450307e7d32-kube-api-access-7jbpq\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:08:51.410302 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:51.410268    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf" event={"ID":"e1116a67-a2e0-4537-9868-e450307e7d32","Type":"ContainerDied","Data":"06e43d15369c53fbb32e53156c4ee487f2a253d24219c5d018f28600dd03ce4d"}
Jul 25 02:08:51.410302 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:51.410297    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"
Jul 25 02:08:51.410769 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:51.410354    2572 scope.go:117] "RemoveContainer" containerID="4aafe11c009886cff0d2fb2137126767e31d4b55300b884df65267111398285b"
Jul 25 02:08:51.431991 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:51.431967    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"]
Jul 25 02:08:51.437098 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:51.437072    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-299af2-774f8f476d-7jzmf"]
Jul 25 02:08:52.352638 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:52.352599    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1116a67-a2e0-4537-9868-e450307e7d32" path="/var/lib/kubelet/pods/e1116a67-a2e0-4537-9868-e450307e7d32/volumes"
Jul 25 02:08:54.887604 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:54.887568    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"]
Jul 25 02:08:54.891103 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:54.891077    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-299af2-hshkj"]
Jul 25 02:08:56.355365 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:56.355312    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de44c725-7082-4f3e-8d4c-a4f18a261fc2" path="/var/lib/kubelet/pods/de44c725-7082-4f3e-8d4c-a4f18a261fc2/volumes"
Jul 25 02:08:56.889344 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:08:56.887050    2572 scope.go:117] "RemoveContainer" containerID="f782dc70d51867a6367b271da3a817f854116de4a8b997c1f4abe8e88128d83d"
Jul 25 02:09:01.446073 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.446039    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"]
Jul 25 02:09:01.446638 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.446259    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy" containerID="cri-o://8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c" gracePeriod=30
Jul 25 02:09:01.622129 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.622090    2572 patch_prober.go:28] interesting pod/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 25 02:09:01.622290 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.622146    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 25 02:09:01.837263 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.837181    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"]
Jul 25 02:09:01.837558 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.837544    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1116a67-a2e0-4537-9868-e450307e7d32" containerName="maas-api"
Jul 25 02:09:01.837604 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.837560    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1116a67-a2e0-4537-9868-e450307e7d32" containerName="maas-api"
Jul 25 02:09:01.837604 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.837568    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de44c725-7082-4f3e-8d4c-a4f18a261fc2" containerName="revoke-keys"
Jul 25 02:09:01.837604 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.837574    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="de44c725-7082-4f3e-8d4c-a4f18a261fc2" containerName="revoke-keys"
Jul 25 02:09:01.837702 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.837649    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="de44c725-7082-4f3e-8d4c-a4f18a261fc2" containerName="revoke-keys"
Jul 25 02:09:01.837702 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.837659    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1116a67-a2e0-4537-9868-e450307e7d32" containerName="maas-api"
Jul 25 02:09:01.842462 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.842445    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"
Jul 25 02:09:01.844778 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.844748    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-88hjh\""
Jul 25 02:09:01.847781 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.847755    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"]
Jul 25 02:09:01.929129 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:01.929100    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a09c243-53c5-40b4-a45a-fc7afb23cef6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8\" (UID: \"7a09c243-53c5-40b4-a45a-fc7afb23cef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"
Jul 25 02:09:02.030132 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:02.030097    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a09c243-53c5-40b4-a45a-fc7afb23cef6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8\" (UID: \"7a09c243-53c5-40b4-a45a-fc7afb23cef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"
Jul 25 02:09:02.030822 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:02.030795    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a09c243-53c5-40b4-a45a-fc7afb23cef6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8\" (UID: \"7a09c243-53c5-40b4-a45a-fc7afb23cef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"
Jul 25 02:09:02.153830 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:02.153798    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"
Jul 25 02:09:02.280311 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:02.280285    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"]
Jul 25 02:09:02.282378 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:09:02.282318    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a09c243_53c5_40b4_a45a_fc7afb23cef6.slice/crio-7f6ad44d13c32260214cae20a45ee49def451aab6f2e1c0ded638d6b752d5e9a WatchSource:0}: Error finding container 7f6ad44d13c32260214cae20a45ee49def451aab6f2e1c0ded638d6b752d5e9a: Status 404 returned error can't find the container with id 7f6ad44d13c32260214cae20a45ee49def451aab6f2e1c0ded638d6b752d5e9a
Jul 25 02:09:02.466235 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:02.466199    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8" event={"ID":"7a09c243-53c5-40b4-a45a-fc7afb23cef6","Type":"ContainerStarted","Data":"ec4bfca17198cf93515ff8518b0ab909da8efaf0c54be15a87b47ed1cf545778"}
Jul 25 02:09:02.466665 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:02.466247    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8" event={"ID":"7a09c243-53c5-40b4-a45a-fc7afb23cef6","Type":"ContainerStarted","Data":"7f6ad44d13c32260214cae20a45ee49def451aab6f2e1c0ded638d6b752d5e9a"}
Jul 25 02:09:02.479755 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:02.479711    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8" podStartSLOduration=1.479697863 podStartE2EDuration="1.479697863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:09:02.479197866 +0000 UTC m=+3426.589007151" watchObservedRunningTime="2026-07-25 02:09:02.479697863 +0000 UTC m=+3426.589507146"
Jul 25 02:09:03.473386 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:03.473315    2572 generic.go:356] "Generic (PLEG): container finished" podID="7a09c243-53c5-40b4-a45a-fc7afb23cef6" containerID="ec4bfca17198cf93515ff8518b0ab909da8efaf0c54be15a87b47ed1cf545778" exitCode=0
Jul 25 02:09:03.473771 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:03.473405    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8" event={"ID":"7a09c243-53c5-40b4-a45a-fc7afb23cef6","Type":"ContainerDied","Data":"ec4bfca17198cf93515ff8518b0ab909da8efaf0c54be15a87b47ed1cf545778"}
Jul 25 02:09:04.606078 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:04.606052    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"
Jul 25 02:09:04.762271 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:04.762195    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a09c243-53c5-40b4-a45a-fc7afb23cef6-maas-api-service-ca\") pod \"7a09c243-53c5-40b4-a45a-fc7afb23cef6\" (UID: \"7a09c243-53c5-40b4-a45a-fc7afb23cef6\") "
Jul 25 02:09:04.762590 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:04.762566    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a09c243-53c5-40b4-a45a-fc7afb23cef6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a09c243-53c5-40b4-a45a-fc7afb23cef6" (UID: "7a09c243-53c5-40b4-a45a-fc7afb23cef6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 02:09:04.863143 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:04.863107    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a09c243-53c5-40b4-a45a-fc7afb23cef6-maas-api-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:05.485690 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:05.485661    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"
Jul 25 02:09:05.485880 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:05.485709    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8" event={"ID":"7a09c243-53c5-40b4-a45a-fc7afb23cef6","Type":"ContainerDied","Data":"7f6ad44d13c32260214cae20a45ee49def451aab6f2e1c0ded638d6b752d5e9a"}
Jul 25 02:09:05.485880 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:05.485764    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f6ad44d13c32260214cae20a45ee49def451aab6f2e1c0ded638d6b752d5e9a"
Jul 25 02:09:06.695998 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.695973    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 02:09:06.779615 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.779544    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-socket\") pod \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") "
Jul 25 02:09:06.779615 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.779586    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-envoy\") pod \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") "
Jul 25 02:09:06.779615 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.779615    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhsjk\" (UniqueName: \"kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-kube-api-access-fhsjk\") pod \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") "
Jul 25 02:09:06.779860 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.779652    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istiod-ca-cert\") pod \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") "
Jul 25 02:09:06.779860 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.779669    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-certs\") pod \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") "
Jul 25 02:09:06.779860 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.779685    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-data\") pod \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") "
Jul 25 02:09:06.779860 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.779714    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-podinfo\") pod \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") "
Jul 25 02:09:06.779860 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.779760    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-token\") pod \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") "
Jul 25 02:09:06.779860 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.779796    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-credential-socket\") pod \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\" (UID: \"ea1388cc-c774-4532-8eea-04eaa8f7bba6\") "
Jul 25 02:09:06.780186 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.779852    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ea1388cc-c774-4532-8eea-04eaa8f7bba6" (UID: "ea1388cc-c774-4532-8eea-04eaa8f7bba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:09:06.780186 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.779986    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ea1388cc-c774-4532-8eea-04eaa8f7bba6" (UID: "ea1388cc-c774-4532-8eea-04eaa8f7bba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:09:06.780186 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.780038    2572 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:06.780186 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.780106    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ea1388cc-c774-4532-8eea-04eaa8f7bba6" (UID: "ea1388cc-c774-4532-8eea-04eaa8f7bba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 02:09:06.780393 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.780223    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ea1388cc-c774-4532-8eea-04eaa8f7bba6" (UID: "ea1388cc-c774-4532-8eea-04eaa8f7bba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:09:06.780729 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.780695    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-data" (OuterVolumeSpecName: "istio-data") pod "ea1388cc-c774-4532-8eea-04eaa8f7bba6" (UID: "ea1388cc-c774-4532-8eea-04eaa8f7bba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:09:06.781993 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.781963    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ea1388cc-c774-4532-8eea-04eaa8f7bba6" (UID: "ea1388cc-c774-4532-8eea-04eaa8f7bba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 25 02:09:06.782101 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.781998    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-kube-api-access-fhsjk" (OuterVolumeSpecName: "kube-api-access-fhsjk") pod "ea1388cc-c774-4532-8eea-04eaa8f7bba6" (UID: "ea1388cc-c774-4532-8eea-04eaa8f7bba6"). InnerVolumeSpecName "kube-api-access-fhsjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:09:06.782250 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.782229    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-token" (OuterVolumeSpecName: "istio-token") pod "ea1388cc-c774-4532-8eea-04eaa8f7bba6" (UID: "ea1388cc-c774-4532-8eea-04eaa8f7bba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:09:06.782250 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.782240    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ea1388cc-c774-4532-8eea-04eaa8f7bba6" (UID: "ea1388cc-c774-4532-8eea-04eaa8f7bba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 25 02:09:06.881026 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.880996    2572 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istiod-ca-cert\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:06.881026 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.881022    2572 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-workload-certs\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:06.881026 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.881031    2572 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-data\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:06.881224 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.881038    2572 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-podinfo\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:06.881224 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.881046    2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-token\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:06.881224 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.881054    2572 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-credential-socket\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:06.881224 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.881063    2572 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea1388cc-c774-4532-8eea-04eaa8f7bba6-istio-envoy\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:06.881224 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:06.881071    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhsjk\" (UniqueName: \"kubernetes.io/projected/ea1388cc-c774-4532-8eea-04eaa8f7bba6-kube-api-access-fhsjk\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:07.497230 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.497197    2572 generic.go:356] "Generic (PLEG): container finished" podID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerID="8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c" exitCode=0
Jul 25 02:09:07.497440 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.497243    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"
Jul 25 02:09:07.497440 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.497275    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" event={"ID":"ea1388cc-c774-4532-8eea-04eaa8f7bba6","Type":"ContainerDied","Data":"8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c"}
Jul 25 02:09:07.497440 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.497342    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm" event={"ID":"ea1388cc-c774-4532-8eea-04eaa8f7bba6","Type":"ContainerDied","Data":"a581483abda701ab94d06678beba944812fc28c640bdd681f645888abbfb1350"}
Jul 25 02:09:07.497440 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.497366    2572 scope.go:117] "RemoveContainer" containerID="8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c"
Jul 25 02:09:07.507217 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.507202    2572 scope.go:117] "RemoveContainer" containerID="0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a"
Jul 25 02:09:07.516088 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.516060    2572 scope.go:117] "RemoveContainer" containerID="8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c"
Jul 25 02:09:07.516391 ip-10-0-135-99 kubenswrapper[2572]: E0725 02:09:07.516361    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c\": container with ID starting with 8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c not found: ID does not exist" containerID="8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c"
Jul 25 02:09:07.516492 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.516402    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c"} err="failed to get container status \"8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c\": rpc error: code = NotFound desc = could not find container \"8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c\": container with ID starting with 8cf8294342898d83fcbbd3b5838c747cdefb4e0cc000ece3552a01aaa2b8220c not found: ID does not exist"
Jul 25 02:09:07.516492 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.516424    2572 scope.go:117] "RemoveContainer" containerID="0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a"
Jul 25 02:09:07.516784 ip-10-0-135-99 kubenswrapper[2572]: E0725 02:09:07.516757    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a\": container with ID starting with 0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a not found: ID does not exist" containerID="0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a"
Jul 25 02:09:07.516895 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.516790    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a"} err="failed to get container status \"0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a\": rpc error: code = NotFound desc = could not find container \"0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a\": container with ID starting with 0aa78cea546011679fca89c4d2d37be2f304b10e0009cb0f936f34003d82206a not found: ID does not exist"
Jul 25 02:09:07.517918 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.517895    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"]
Jul 25 02:09:07.521394 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:07.521371    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-299af2-openshift-default-7fd54df7b8-wbrtm"]
Jul 25 02:09:08.358626 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:08.358534    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" path="/var/lib/kubelet/pods/ea1388cc-c774-4532-8eea-04eaa8f7bba6/volumes"
Jul 25 02:09:12.353882 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.353848    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"]
Jul 25 02:09:12.354352 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.354068    2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf" podUID="80363b91-fbb0-458e-9241-d3120ee8b6e0" containerName="maas-api" containerID="cri-o://342b042817a2aa6676e4d36dca2c87658ab38c4aa4ea0f535c3ad93fb4d48966" gracePeriod=30
Jul 25 02:09:12.527365 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.527306    2572 generic.go:356] "Generic (PLEG): container finished" podID="80363b91-fbb0-458e-9241-d3120ee8b6e0" containerID="342b042817a2aa6676e4d36dca2c87658ab38c4aa4ea0f535c3ad93fb4d48966" exitCode=0
Jul 25 02:09:12.527490 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.527365    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf" event={"ID":"80363b91-fbb0-458e-9241-d3120ee8b6e0","Type":"ContainerDied","Data":"342b042817a2aa6676e4d36dca2c87658ab38c4aa4ea0f535c3ad93fb4d48966"}
Jul 25 02:09:12.592021 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.591997    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 02:09:12.726560 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.726535    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80363b91-fbb0-458e-9241-d3120ee8b6e0-maas-api-tls\") pod \"80363b91-fbb0-458e-9241-d3120ee8b6e0\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") "
Jul 25 02:09:12.726739 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.726569    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80363b91-fbb0-458e-9241-d3120ee8b6e0-openshift-service-ca\") pod \"80363b91-fbb0-458e-9241-d3120ee8b6e0\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") "
Jul 25 02:09:12.726739 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.726614    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgtsk\" (UniqueName: \"kubernetes.io/projected/80363b91-fbb0-458e-9241-d3120ee8b6e0-kube-api-access-jgtsk\") pod \"80363b91-fbb0-458e-9241-d3120ee8b6e0\" (UID: \"80363b91-fbb0-458e-9241-d3120ee8b6e0\") "
Jul 25 02:09:12.727001 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.726973    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80363b91-fbb0-458e-9241-d3120ee8b6e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "80363b91-fbb0-458e-9241-d3120ee8b6e0" (UID: "80363b91-fbb0-458e-9241-d3120ee8b6e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 25 02:09:12.728773 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.728740    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80363b91-fbb0-458e-9241-d3120ee8b6e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "80363b91-fbb0-458e-9241-d3120ee8b6e0" (UID: "80363b91-fbb0-458e-9241-d3120ee8b6e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 25 02:09:12.728866 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.728817    2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80363b91-fbb0-458e-9241-d3120ee8b6e0-kube-api-access-jgtsk" (OuterVolumeSpecName: "kube-api-access-jgtsk") pod "80363b91-fbb0-458e-9241-d3120ee8b6e0" (UID: "80363b91-fbb0-458e-9241-d3120ee8b6e0"). InnerVolumeSpecName "kube-api-access-jgtsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 25 02:09:12.827907 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.827882    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgtsk\" (UniqueName: \"kubernetes.io/projected/80363b91-fbb0-458e-9241-d3120ee8b6e0-kube-api-access-jgtsk\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:12.827907 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.827904    2572 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80363b91-fbb0-458e-9241-d3120ee8b6e0-maas-api-tls\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:12.827907 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:12.827913    2572 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80363b91-fbb0-458e-9241-d3120ee8b6e0-openshift-service-ca\") on node \"ip-10-0-135-99.ec2.internal\" DevicePath \"\""
Jul 25 02:09:13.534467 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:13.534438    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"
Jul 25 02:09:13.534467 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:13.534453    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf" event={"ID":"80363b91-fbb0-458e-9241-d3120ee8b6e0","Type":"ContainerDied","Data":"b0d200e82c5a2e18fb9cf80c39b4158c22c302e102bb99691c65fce410ef93b3"}
Jul 25 02:09:13.534930 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:13.534507    2572 scope.go:117] "RemoveContainer" containerID="342b042817a2aa6676e4d36dca2c87658ab38c4aa4ea0f535c3ad93fb4d48966"
Jul 25 02:09:13.555990 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:13.555962    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"]
Jul 25 02:09:13.560018 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:13.559987    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ab0ae6-796cd79695-n6zsf"]
Jul 25 02:09:14.353111 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:14.353075    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80363b91-fbb0-458e-9241-d3120ee8b6e0" path="/var/lib/kubelet/pods/80363b91-fbb0-458e-9241-d3120ee8b6e0/volumes"
Jul 25 02:09:16.904912 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:16.904875    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"]
Jul 25 02:09:16.907144 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:16.907121    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-ab0ae6-vchw8"]
Jul 25 02:09:18.353065 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:18.353027    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a09c243-53c5-40b4-a45a-fc7afb23cef6" path="/var/lib/kubelet/pods/7a09c243-53c5-40b4-a45a-fc7afb23cef6/volumes"
Jul 25 02:09:24.118969 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:24.118935    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cf5d9ddd-x6svt_6054bfae-2e7a-4f2c-a37c-a35845d024d6/authorino/0.log"
Jul 25 02:09:30.670148 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:30.670116    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7587ff6c45-nnbmn_755b8adb-c446-4f61-b3b4-18e9b080e521/copy-manifests/0.log"
Jul 25 02:09:30.677127 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:30.677103    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7587ff6c45-nnbmn_755b8adb-c446-4f61-b3b4-18e9b080e521/manager/0.log"
Jul 25 02:09:31.037365 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:31.037270    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d6c5bd49-js487_774f4ffa-4f01-49f0-ad82-49289fc182d8/maas-api/0.log"
Jul 25 02:09:31.155248 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:31.155216    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29749050-pqr5s_3e0cd0ad-e1ed-4788-b244-16dface55598/cleanup/0.log"
Jul 25 02:09:31.268925 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:31.268897    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29749065-6wxnv_99541586-a0e1-4f0f-8ce4-f8ee7c8cd843/cleanup/0.log"
Jul 25 02:09:31.385611 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:31.385532    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29749080-x4lsk_23ddb903-a328-4b87-85a1-cbaa539b16da/cleanup/0.log"
Jul 25 02:09:31.502013 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:31.501986    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-lshxp_a300f37a-5848-4633-bb57-36bd7c2c5b3b/postgres/0.log"
Jul 25 02:09:32.320842 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.320819    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582_05aba405-3737-42e4-9d71-d24fd75f885b/pull/0.log"
Jul 25 02:09:32.326657 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.326635    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582_05aba405-3737-42e4-9d71-d24fd75f885b/extract/0.log"
Jul 25 02:09:32.332525 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.332498    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582_05aba405-3737-42e4-9d71-d24fd75f885b/util/0.log"
Jul 25 02:09:32.454641 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.454617    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm_402756f6-2e2e-46e9-bd2e-2f9f0ed469d0/util/0.log"
Jul 25 02:09:32.460278 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.460259    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm_402756f6-2e2e-46e9-bd2e-2f9f0ed469d0/pull/0.log"
Jul 25 02:09:32.465582 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.465567    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm_402756f6-2e2e-46e9-bd2e-2f9f0ed469d0/extract/0.log"
Jul 25 02:09:32.581276 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.581199    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm_2cad4dfe-1404-4c88-9a78-cfaac6b38eb6/util/0.log"
Jul 25 02:09:32.587458 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.587440    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm_2cad4dfe-1404-4c88-9a78-cfaac6b38eb6/pull/0.log"
Jul 25 02:09:32.593366 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.593350    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm_2cad4dfe-1404-4c88-9a78-cfaac6b38eb6/extract/0.log"
Jul 25 02:09:32.712747 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.712721    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h_ab48424a-6670-4bbc-885e-843199410862/extract/0.log"
Jul 25 02:09:32.718268 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.718249    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h_ab48424a-6670-4bbc-885e-843199410862/util/0.log"
Jul 25 02:09:32.724176 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.724158    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h_ab48424a-6670-4bbc-885e-843199410862/pull/0.log"
Jul 25 02:09:32.845685 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:32.845625    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cf5d9ddd-x6svt_6054bfae-2e7a-4f2c-a37c-a35845d024d6/authorino/0.log"
Jul 25 02:09:33.204759 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:33.204735    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-hk4cj_0336442f-1ed9-439f-835b-38f87c1657cb/kuadrant-console-plugin/0.log"
Jul 25 02:09:34.282113 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:34.282087    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789f49c94d-6988f_13f5b243-a80b-40d5-90f5-d920044e35e2/kube-auth-proxy/0.log"
Jul 25 02:09:35.363046 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:35.363018    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jl5fb_3a58ad35-21f8-46d1-aee5-fc8267eba096/main/0.log"
Jul 25 02:09:41.250747 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.250715    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qpj2w/must-gather-gn9hn"]
Jul 25 02:09:41.251169 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251127    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80363b91-fbb0-458e-9241-d3120ee8b6e0" containerName="maas-api"
Jul 25 02:09:41.251169 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251148    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="80363b91-fbb0-458e-9241-d3120ee8b6e0" containerName="maas-api"
Jul 25 02:09:41.251169 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251167    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy"
Jul 25 02:09:41.251403 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251177    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy"
Jul 25 02:09:41.251403 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251215    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09c243-53c5-40b4-a45a-fc7afb23cef6" containerName="revoke-keys"
Jul 25 02:09:41.251403 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251225    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09c243-53c5-40b4-a45a-fc7afb23cef6" containerName="revoke-keys"
Jul 25 02:09:41.251403 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251311    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a09c243-53c5-40b4-a45a-fc7afb23cef6" containerName="revoke-keys"
Jul 25 02:09:41.251403 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251341    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="80363b91-fbb0-458e-9241-d3120ee8b6e0" containerName="maas-api"
Jul 25 02:09:41.251403 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251369    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy"
Jul 25 02:09:41.251403 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251377    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy"
Jul 25 02:09:41.251649 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251454    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy"
Jul 25 02:09:41.251649 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.251461    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea1388cc-c774-4532-8eea-04eaa8f7bba6" containerName="istio-proxy"
Jul 25 02:09:41.254416 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.254394    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qpj2w/must-gather-gn9hn"
Jul 25 02:09:41.257219 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.257192    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qpj2w\"/\"default-dockercfg-b5m6f\""
Jul 25 02:09:41.257450 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.257436    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qpj2w\"/\"openshift-service-ca.crt\""
Jul 25 02:09:41.258214 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.258195    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qpj2w\"/\"kube-root-ca.crt\""
Jul 25 02:09:41.267097 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.267073    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qpj2w/must-gather-gn9hn"]
Jul 25 02:09:41.358213 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.358179    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9eaa47d6-d98d-4b5a-843c-304da63cb82a-must-gather-output\") pod \"must-gather-gn9hn\" (UID: \"9eaa47d6-d98d-4b5a-843c-304da63cb82a\") " pod="openshift-must-gather-qpj2w/must-gather-gn9hn"
Jul 25 02:09:41.358213 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.358211    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbr8v\" (UniqueName: \"kubernetes.io/projected/9eaa47d6-d98d-4b5a-843c-304da63cb82a-kube-api-access-tbr8v\") pod \"must-gather-gn9hn\" (UID: \"9eaa47d6-d98d-4b5a-843c-304da63cb82a\") " pod="openshift-must-gather-qpj2w/must-gather-gn9hn"
Jul 25 02:09:41.458990 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.458948    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9eaa47d6-d98d-4b5a-843c-304da63cb82a-must-gather-output\") pod \"must-gather-gn9hn\" (UID: \"9eaa47d6-d98d-4b5a-843c-304da63cb82a\") " pod="openshift-must-gather-qpj2w/must-gather-gn9hn"
Jul 25 02:09:41.459187 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.459007    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbr8v\" (UniqueName: \"kubernetes.io/projected/9eaa47d6-d98d-4b5a-843c-304da63cb82a-kube-api-access-tbr8v\") pod \"must-gather-gn9hn\" (UID: \"9eaa47d6-d98d-4b5a-843c-304da63cb82a\") " pod="openshift-must-gather-qpj2w/must-gather-gn9hn"
Jul 25 02:09:41.459342 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.459304    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9eaa47d6-d98d-4b5a-843c-304da63cb82a-must-gather-output\") pod \"must-gather-gn9hn\" (UID: \"9eaa47d6-d98d-4b5a-843c-304da63cb82a\") " pod="openshift-must-gather-qpj2w/must-gather-gn9hn"
Jul 25 02:09:41.482841 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.482811    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbr8v\" (UniqueName: \"kubernetes.io/projected/9eaa47d6-d98d-4b5a-843c-304da63cb82a-kube-api-access-tbr8v\") pod \"must-gather-gn9hn\" (UID: \"9eaa47d6-d98d-4b5a-843c-304da63cb82a\") " pod="openshift-must-gather-qpj2w/must-gather-gn9hn"
Jul 25 02:09:41.563077 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.562984    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qpj2w/must-gather-gn9hn"
Jul 25 02:09:41.695763 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.695731    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qpj2w/must-gather-gn9hn"]
Jul 25 02:09:41.698229 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:09:41.698204    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eaa47d6_d98d_4b5a_843c_304da63cb82a.slice/crio-e619cd9d0f21b8eca42f3ca5b877da8da57a2bf29e4e11d5e500551c5aa8e473 WatchSource:0}: Error finding container e619cd9d0f21b8eca42f3ca5b877da8da57a2bf29e4e11d5e500551c5aa8e473: Status 404 returned error can't find the container with id e619cd9d0f21b8eca42f3ca5b877da8da57a2bf29e4e11d5e500551c5aa8e473
Jul 25 02:09:41.700503 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:41.700484    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 25 02:09:42.696158 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:42.696118    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qpj2w/must-gather-gn9hn" event={"ID":"9eaa47d6-d98d-4b5a-843c-304da63cb82a","Type":"ContainerStarted","Data":"461b3f28629c3589e5e3fae35549c4ecd67f041600c08fc1e48636b3d823144b"}
Jul 25 02:09:42.696626 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:42.696170    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qpj2w/must-gather-gn9hn" event={"ID":"9eaa47d6-d98d-4b5a-843c-304da63cb82a","Type":"ContainerStarted","Data":"5dcee945faad533937e46709660d572ba5eb353f7cc31bd8fa59ae2ab9345378"}
Jul 25 02:09:42.696626 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:42.696188    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qpj2w/must-gather-gn9hn" event={"ID":"9eaa47d6-d98d-4b5a-843c-304da63cb82a","Type":"ContainerStarted","Data":"e619cd9d0f21b8eca42f3ca5b877da8da57a2bf29e4e11d5e500551c5aa8e473"}
Jul 25 02:09:42.712979 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:42.712920    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qpj2w/must-gather-gn9hn" podStartSLOduration=0.939864006 podStartE2EDuration="1.712897804s" totalImagesPullingTime="773.033798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 02:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:09:42.710510714 +0000 UTC m=+3466.820320023" watchObservedRunningTime="2026-07-25 02:09:42.712897804 +0000 UTC m=+3466.822707089"
Jul 25 02:09:43.963311 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:43.963280    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hkv52_6c8fc4b5-4d66-4ed8-948b-a9a57931649e/global-pull-secret-syncer/0.log"
Jul 25 02:09:44.080753 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:44.080717    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rk9b7_d9d7e4de-3c40-43fd-a2c1-67117b28520e/konnectivity-agent/0.log"
Jul 25 02:09:44.177832 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:44.177798    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-99.ec2.internal_47c77adca673d868115fa74ace1d7b3f/haproxy/0.log"
Jul 25 02:09:48.342601 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.342562    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582_05aba405-3737-42e4-9d71-d24fd75f885b/extract/0.log"
Jul 25 02:09:48.358025 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.357992    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582_05aba405-3737-42e4-9d71-d24fd75f885b/util/0.log"
Jul 25 02:09:48.378317 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.378284    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7594n582_05aba405-3737-42e4-9d71-d24fd75f885b/pull/0.log"
Jul 25 02:09:48.405903 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.405871    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm_402756f6-2e2e-46e9-bd2e-2f9f0ed469d0/extract/0.log"
Jul 25 02:09:48.428537 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.428504    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm_402756f6-2e2e-46e9-bd2e-2f9f0ed469d0/util/0.log"
Jul 25 02:09:48.448145 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.448104    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0t7lfm_402756f6-2e2e-46e9-bd2e-2f9f0ed469d0/pull/0.log"
Jul 25 02:09:48.474662 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.474628    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm_2cad4dfe-1404-4c88-9a78-cfaac6b38eb6/extract/0.log"
Jul 25 02:09:48.496542 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.496510    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm_2cad4dfe-1404-4c88-9a78-cfaac6b38eb6/util/0.log"
Jul 25 02:09:48.520042 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.520014    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73lpmnm_2cad4dfe-1404-4c88-9a78-cfaac6b38eb6/pull/0.log"
Jul 25 02:09:48.545683 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.545651    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h_ab48424a-6670-4bbc-885e-843199410862/extract/0.log"
Jul 25 02:09:48.572650 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.572619    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h_ab48424a-6670-4bbc-885e-843199410862/util/0.log"
Jul 25 02:09:48.597008 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:48.596915    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cd77h_ab48424a-6670-4bbc-885e-843199410862/pull/0.log"
Jul 25 02:09:49.059056 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:49.058877    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cf5d9ddd-x6svt_6054bfae-2e7a-4f2c-a37c-a35845d024d6/authorino/0.log"
Jul 25 02:09:49.150642 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:49.150602    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-hk4cj_0336442f-1ed9-439f-835b-38f87c1657cb/kuadrant-console-plugin/0.log"
Jul 25 02:09:51.222591 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.222559    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-77n8k_b33f1931-861a-4b78-b7d3-5433ad272a0f/kube-state-metrics/0.log"
Jul 25 02:09:51.244864 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.244838    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-77n8k_b33f1931-861a-4b78-b7d3-5433ad272a0f/kube-rbac-proxy-main/0.log"
Jul 25 02:09:51.278379 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.278354    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-77n8k_b33f1931-861a-4b78-b7d3-5433ad272a0f/kube-rbac-proxy-self/0.log"
Jul 25 02:09:51.543858 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.543761    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z2kt7_4d84a197-6a94-48f6-825e-16181f3b44d5/node-exporter/0.log"
Jul 25 02:09:51.574873 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.574837    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z2kt7_4d84a197-6a94-48f6-825e-16181f3b44d5/kube-rbac-proxy/0.log"
Jul 25 02:09:51.596171 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.596142    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z2kt7_4d84a197-6a94-48f6-825e-16181f3b44d5/init-textfile/0.log"
Jul 25 02:09:51.636849 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.636799    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6fqm5_5ff29538-913e-4789-bc77-ba85057fb62c/kube-rbac-proxy-main/0.log"
Jul 25 02:09:51.659220 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.659171    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6fqm5_5ff29538-913e-4789-bc77-ba85057fb62c/kube-rbac-proxy-self/0.log"
Jul 25 02:09:51.681633 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.681602    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-6fqm5_5ff29538-913e-4789-bc77-ba85057fb62c/openshift-state-metrics/0.log"
Jul 25 02:09:51.909077 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.909046    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-rc8fx_635ecd8f-a367-43ba-82bd-e81cf39131b7/prometheus-operator/0.log"
Jul 25 02:09:51.927030 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.926999    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-rc8fx_635ecd8f-a367-43ba-82bd-e81cf39131b7/kube-rbac-proxy/0.log"
Jul 25 02:09:51.955882 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:51.955854    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-hjcfs_3239601d-ad90-41f4-b504-375eb1a61156/prometheus-operator-admission-webhook/0.log"
Jul 25 02:09:54.263513 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.263470    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"]
Jul 25 02:09:54.269432 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.269404    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.275284 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.275259    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"]
Jul 25 02:09:54.393280 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.393240    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-proc\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.393577 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.393556    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-lib-modules\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.393788 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.393771    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-podres\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.393940 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.393925    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-sys\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.394113 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.394093    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzp27\" (UniqueName: \"kubernetes.io/projected/69161f4f-f7e1-4ee7-a353-93c72d188bba-kube-api-access-hzp27\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.495368 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.495299    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-podres\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.495565 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.495385    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-sys\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.495565 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.495418    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzp27\" (UniqueName: \"kubernetes.io/projected/69161f4f-f7e1-4ee7-a353-93c72d188bba-kube-api-access-hzp27\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.495565 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.495476    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-proc\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.495565 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.495503    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-lib-modules\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.495565 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.495507    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-podres\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.495565 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.495522    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-sys\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.495908 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.495614    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-proc\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.495908 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.495681    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69161f4f-f7e1-4ee7-a353-93c72d188bba-lib-modules\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.503623 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.503571    2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzp27\" (UniqueName: \"kubernetes.io/projected/69161f4f-f7e1-4ee7-a353-93c72d188bba-kube-api-access-hzp27\") pod \"perf-node-gather-daemonset-z6c7h\" (UID: \"69161f4f-f7e1-4ee7-a353-93c72d188bba\") " pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.582231 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.582140    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:54.804345 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:54.802021    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"]
Jul 25 02:09:54.818402 ip-10-0-135-99 kubenswrapper[2572]: W0725 02:09:54.818359    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod69161f4f_f7e1_4ee7_a353_93c72d188bba.slice/crio-a0106f2d297017fa433ae96470b90985391adda4b501ec6a222e772ee9bde250 WatchSource:0}: Error finding container a0106f2d297017fa433ae96470b90985391adda4b501ec6a222e772ee9bde250: Status 404 returned error can't find the container with id a0106f2d297017fa433ae96470b90985391adda4b501ec6a222e772ee9bde250
Jul 25 02:09:55.802680 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:55.802637    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h" event={"ID":"69161f4f-f7e1-4ee7-a353-93c72d188bba","Type":"ContainerStarted","Data":"9bfbaff5384e44d66318ef3c124cd9a89b822ede9d037e772c493aaf03dd8987"}
Jul 25 02:09:55.803176 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:55.802709    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h" event={"ID":"69161f4f-f7e1-4ee7-a353-93c72d188bba","Type":"ContainerStarted","Data":"a0106f2d297017fa433ae96470b90985391adda4b501ec6a222e772ee9bde250"}
Jul 25 02:09:55.803176 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:55.802938    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:09:55.821103 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:55.821054    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h" podStartSLOduration=1.821036838 podStartE2EDuration="1.821036838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 02:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 02:09:55.817585583 +0000 UTC m=+3479.927394864" watchObservedRunningTime="2026-07-25 02:09:55.821036838 +0000 UTC m=+3479.930846122"
Jul 25 02:09:56.327292 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:56.327258    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2bkbl_183e983e-4d28-47d2-be8e-36cfd1293dae/dns/0.log"
Jul 25 02:09:56.345833 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:56.345806    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2bkbl_183e983e-4d28-47d2-be8e-36cfd1293dae/kube-rbac-proxy/0.log"
Jul 25 02:09:56.429560 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:56.429532    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fwsm2_0889cc33-fc59-4e79-a109-6e98bf53cc58/dns-node-resolver/0.log"
Jul 25 02:09:56.973927 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:56.973893    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-559c56d4b8-fnljn_4a2ab7fb-d629-472f-8fd6-5c6862caa9a3/registry/0.log"
Jul 25 02:09:57.017513 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:57.017487    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jd9wg_373f9d31-a96b-4a50-b460-0716a102570f/node-ca/0.log"
Jul 25 02:09:58.107464 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:58.107437    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789f49c94d-6988f_13f5b243-a80b-40d5-90f5-d920044e35e2/kube-auth-proxy/0.log"
Jul 25 02:09:59.084630 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:59.084597    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-x62hp_cc0d00f8-2138-4cf2-98bf-d1898e41ddd1/serve-healthcheck-canary/0.log"
Jul 25 02:09:59.615796 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:59.615767    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-js2ft_4dff2d71-7a7b-4918-b55c-614b1a8d2065/kube-rbac-proxy/0.log"
Jul 25 02:09:59.635543 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:59.635515    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-js2ft_4dff2d71-7a7b-4918-b55c-614b1a8d2065/exporter/0.log"
Jul 25 02:09:59.654229 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:09:59.654205    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-js2ft_4dff2d71-7a7b-4918-b55c-614b1a8d2065/extractor/0.log"
Jul 25 02:10:01.823508 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:01.823473    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qpj2w/perf-node-gather-daemonset-z6c7h"
Jul 25 02:10:02.379910 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:02.379875    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7587ff6c45-nnbmn_755b8adb-c446-4f61-b3b4-18e9b080e521/manager/0.log"
Jul 25 02:10:02.398951 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:02.398923    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7587ff6c45-nnbmn_755b8adb-c446-4f61-b3b4-18e9b080e521/copy-manifests/0.log"
Jul 25 02:10:03.789995 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:03.789949    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6b87879b5d-k6bs4_cdae8040-882c-4084-bee8-b04b30990873/manager/0.log"
Jul 25 02:10:03.838985 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:03.838935    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9xxwh_598247d6-3791-4ff8-a3e4-a15fdc72b323/openshift-lws-operator/0.log"
Jul 25 02:10:04.743654 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:04.743573    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789f49c94d-6988f_13f5b243-a80b-40d5-90f5-d920044e35e2/kube-auth-proxy/0.log"
Jul 25 02:10:06.429889 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:06.429844    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789f49c94d-6988f_13f5b243-a80b-40d5-90f5-d920044e35e2/kube-auth-proxy/0.log"
Jul 25 02:10:10.174824 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:10.174801    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6nrz_036736e7-e135-4c08-be7a-8ac7d1975cdf/kube-multus-additional-cni-plugins/0.log"
Jul 25 02:10:10.192948 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:10.192926    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6nrz_036736e7-e135-4c08-be7a-8ac7d1975cdf/egress-router-binary-copy/0.log"
Jul 25 02:10:10.210689 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:10.210668    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6nrz_036736e7-e135-4c08-be7a-8ac7d1975cdf/cni-plugins/0.log"
Jul 25 02:10:10.228513 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:10.228495    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6nrz_036736e7-e135-4c08-be7a-8ac7d1975cdf/bond-cni-plugin/0.log"
Jul 25 02:10:10.249105 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:10.249081    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6nrz_036736e7-e135-4c08-be7a-8ac7d1975cdf/routeoverride-cni/0.log"
Jul 25 02:10:10.267533 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:10.267509    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6nrz_036736e7-e135-4c08-be7a-8ac7d1975cdf/whereabouts-cni-bincopy/0.log"
Jul 25 02:10:10.285315 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:10.285292    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-k6nrz_036736e7-e135-4c08-be7a-8ac7d1975cdf/whereabouts-cni/0.log"
Jul 25 02:10:10.615378 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:10.615292    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bt57h_166e0d16-c9f0-4993-b0de-e1cc91fd993a/kube-multus/0.log"
Jul 25 02:10:10.734112 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:10.734086    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nmt2v_1f0a939b-732b-4313-b0f9-2ce5feb9cc5c/network-metrics-daemon/0.log"
Jul 25 02:10:10.749011 ip-10-0-135-99 kubenswrapper[2572]: I0725 02:10:10.748985    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nmt2v_1f0a939b-732b-4313-b0f9-2ce5feb9cc5c/kube-rbac-proxy/0.log"