Jul 23 21:49:40.838401 ip-10-0-135-91 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 23 21:49:40.838535 ip-10-0-135-91 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 23 21:49:40.838551 ip-10-0-135-91 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 23 21:49:40.838944 ip-10-0-135-91 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 23 21:49:50.409871 ip-10-0-135-91 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Jul 23 21:49:50.409887 ip-10-0-135-91 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f978816095b746b5ac71a81707c128b4 --
Jul 23 21:51:44.491225 ip-10-0-135-91 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 21:51:44.873811 ip-10-0-135-91 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 21:51:44.873811 ip-10-0-135-91 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 21:51:44.873811 ip-10-0-135-91 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 21:51:44.873811 ip-10-0-135-91 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 21:51:44.873811 ip-10-0-135-91 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 21:51:44.875534 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.875439    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 21:51:44.881951 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881924    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 21:51:44.881951 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881950    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 21:51:44.881951 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881954    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881957    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881960    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881963    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881966    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881968    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881971    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881974    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881977    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881981    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881986    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881989    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881992    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881994    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.881998    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882001    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882005    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882009    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882012    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 21:51:44.882054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882022    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882025    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882027    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882030    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882032    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882035    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882038    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882040    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882042    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882045    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882048    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882050    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882053    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882055    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882058    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882061    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882063    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882066    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882069    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882072    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 21:51:44.882507 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882074    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882078    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882080    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882083    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882085    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882087    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882090    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882092    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882096    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882098    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882100    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882103    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882105    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882108    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882112    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882115    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882118    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882120    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882123    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882125    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 21:51:44.883023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882128    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882131    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882134    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882136    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882138    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882141    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882143    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882146    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882149    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882151    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882154    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882156    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882159    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882161    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882164    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882166    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882168    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882171    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882173    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882175    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 21:51:44.883508 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882180    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882183    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882185    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882187    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882190    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882194    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882199    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882202    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882204    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882207    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882209    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882212    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882214    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882217    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882219    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882222    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882224    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.882227    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883030    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 21:51:44.884013 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883038    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883042    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883045    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883047    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883050    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883053    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883056    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883058    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883061    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883063    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883066    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883069    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883071    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883074    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883076    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883079    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883082    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883084    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883087    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 21:51:44.884539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883091    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883095    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883100    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883102    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883105    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883108    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883110    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883113    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883115    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883118    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883120    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883122    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883125    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883128    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883130    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883132    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883135    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883138    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883140    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883143    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 21:51:44.885062 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883145    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883147    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883150    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883152    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883155    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883157    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883160    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883163    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883166    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883168    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883171    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883173    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883176    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883179    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883183    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883185    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883188    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883190    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883193    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883195    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 21:51:44.885538 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883198    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883200    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883204    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883207    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883210    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883213    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883215    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883218    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883220    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883222    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883225    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883228    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883230    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883233    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883235    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883238    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883240    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883242    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883245    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883248    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 21:51:44.886054 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883251    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883253    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883256    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883258    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883262    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883265    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883267    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883270    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883275    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883278    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883281    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883284    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883286    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883289    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883292    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883295    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883297    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883300    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883303    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 21:51:44.886539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883748    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883756    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883759    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883762    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883765    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883767    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883772    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883775    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883778    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883781    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883784    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883786    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883789    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883791    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883794    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883797    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883799    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883802    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883805    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 21:51:44.887017 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883808    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883811    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883814    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883817    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883819    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883821    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883824    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883826    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883829    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883831    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883834    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883836    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883838    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883841    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883843    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883846    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883848    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883850    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883853    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883855    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883874    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 21:51:44.887484 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883877    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883879    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883883    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883886    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883889    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883891    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883894    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883896    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883899    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883902    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883905    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883908    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883911    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883914    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883917    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883919    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883922    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883924    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883929    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 21:51:44.888009 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883931    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883934    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883937    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883941    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883944    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883946    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883949    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883951    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883953    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883956    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883958    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883961    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883964    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883966    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883969    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883971    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883974    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883976    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883982    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 21:51:44.888515 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883984    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883987    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883989    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883992    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883994    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.883997    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884000    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884002    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884005    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884008    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884011    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884013    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884016    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884019    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884022    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884025    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884027    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884030    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884032    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884035    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 21:51:44.889042 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884037    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884113    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884120    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884126    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884131    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884135    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884138    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884142    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884146    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884149    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884152    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884157    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884160    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884163    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884166    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884170    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884173    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884184    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884187    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884194    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884197    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884200    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884203    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884207    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 21:51:44.889519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884211    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884214    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884217    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884220    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884223    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884226    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884229    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884232    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884235    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884239    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884242    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884245    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884248    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884251    2579 flags.go:64] FLAG: --enable-server="true"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884254    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884258    2579 flags.go:64] FLAG: --event-burst="100"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884262    2579 flags.go:64] FLAG: --event-qps="50"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884264    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884268    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884272    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884276    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884279    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884282    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884285    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884288    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 21:51:44.890106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884291    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884294    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884297    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884300    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884303    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884306    2579 flags.go:64] FLAG: --feature-gates=""
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884310    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884314    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884317    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884319    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884323    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884326    2579 flags.go:64] FLAG: --help="false"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884328    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-135-91.ec2.internal"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884332    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884351    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884355    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884359    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884363    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884365    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884368    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884371    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884374    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884379    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884382    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 21:51:44.890732 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884385    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884388    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884392    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884395    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884398    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884401    2579 flags.go:64] FLAG: --lock-file=""
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884404    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884407    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884410    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884415    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884418    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884421    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884424    2579 flags.go:64] FLAG: --logging-format="text"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884426    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884430    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884433    2579 flags.go:64] FLAG: --manifest-url=""
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884436    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884441    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884444    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884450    2579 flags.go:64] FLAG: --max-pods="110"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884453    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884456    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884459    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884461    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884465    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 21:51:44.891340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884468    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884470    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884477    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884481    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884483    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884488    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884491    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884496    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884499    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884502    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884506    2579 flags.go:64] FLAG: --port="10250"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884509    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884512    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e3f1b3fec2e23f3c"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884516    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884519    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884522    2579 flags.go:64] FLAG: --register-node="true"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884525    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884529    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884532    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884534    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884537    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884541    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884544    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884548    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884551    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 21:51:44.892005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884553    2579 flags.go:64] FLAG: --runonce="false"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884556    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884559    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884562    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884565    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884567    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884570    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884573    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884576    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884579    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884582    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884584    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884592    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884595    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884598    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884601    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884606    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884611    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884614    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884619    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884622    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884624    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884627    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884630    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884633    2579 flags.go:64] FLAG: --v="2"
Jul 23 21:51:44.892659 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884637    2579 flags.go:64] FLAG: --version="false"
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884641    2579 flags.go:64] FLAG: --vmodule=""
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884645    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.884648    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884778    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884783    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884786    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884789    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884792    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884794    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884797    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884800    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884802    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884806    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884810    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884813    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884816    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884820    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884825    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 21:51:44.893318 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884830    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884833    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884835    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884838    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884841    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884845    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884848    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884850    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884853    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884855    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884871    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884875    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884877    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884880    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884883    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884886    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884888    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884891    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884894    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884897    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 21:51:44.894023 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884900    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884902    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884905    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884907    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884910    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884912    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884915    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884918    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884920    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884923    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884925    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884928    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884932    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884934    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884937    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884939    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884942    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884946    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884949    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884951    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 21:51:44.894540 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884954    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884957    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884959    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884962    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884964    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884967    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884969    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884971    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884974    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884976    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884979    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884981    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884990    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884993    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884996    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.884998    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885000    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885003    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885006    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885008    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 21:51:44.895056 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885011    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885013    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885016    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885018    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885022    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885025    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885028    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885030    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885032    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885036    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885039    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885041    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885044    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885046    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885049    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885051    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885054    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885056    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885059    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885061    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 21:51:44.895546 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885064    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885066    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885069    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.885072    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.885781    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.894685    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.894707    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.894729    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.894735    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894801    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894808    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894812    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894815    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894818    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 21:51:44.896077 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894820    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894824    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894826    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894829    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894831    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894834    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894836    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894839    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894841    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894844    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894847    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894850    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894852    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894855    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894877    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894880    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894883    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894885    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894888    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894891    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 21:51:44.896442 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894894    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894896    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894908    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894911    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894914    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894917    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894920    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894924    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894927    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894930    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894933    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894936    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894938    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894941    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894944    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894946    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894948    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894951    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894954    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 21:51:44.896962 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894956    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894959    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894962    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894964    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894967    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894969    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894973    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894976    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894979    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894982    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894984    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894986    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894989    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894991    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894994    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894996    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.894999    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895003    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895006    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895008    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 21:51:44.897434 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895011    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895014    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895016    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895019    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895021    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895024    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895027    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895029    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895031    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895034    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895037    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895039    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895042    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895045    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895047    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895050    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895053    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895056    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895058    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 21:51:44.897954 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895061    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895064    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895067    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895069    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895072    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895075    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895077    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895079    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895082    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895085    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895088    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895090    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895093    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895095    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895098    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 21:51:44.898424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895101    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.895107    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895234    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895240    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895243    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895246    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895249    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895251    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895254    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895257    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895259    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895262    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895265    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895267    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895270    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 21:51:44.898816 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895273    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895276    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895278    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895281    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895284    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895287    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895289    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895292    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895294    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895297    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895300    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895302    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895305    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895308    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895312    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895315    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895317    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895320    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895322    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 21:51:44.899221 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895324    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895327    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895330    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895332    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895334    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895336    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895339    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895341    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895344    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895346    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895348    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895351    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895353    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895356    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895358    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895361    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895363    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895366    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895369    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895371    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 21:51:44.899691 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895374    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895377    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895379    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895381    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895384    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895387    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895389    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895392    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895394    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895397    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895399    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895402    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895404    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895406    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895409    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895411    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895414    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895416    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895418    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895421    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 21:51:44.900705 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895424    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895426    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895429    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895431    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895434    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895436    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895440    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895442    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895446    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895449    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895453    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895456    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895459    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895462    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895464    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895466    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895469    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895472    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895474    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 21:51:44.901305 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895476    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895479    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895481    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895484    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895486    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895489    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895491    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:44.895494    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.895499    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.895629    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.898175    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.899653    2579 server.go:1019] "Starting client certificate rotation"
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.899775    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 21:51:44.901787 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.899824    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 21:51:44.903587 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.903564    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 21:51:44.908248 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.908217    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 21:51:44.919641 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.919614    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 23 21:51:44.922303 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.922287    2579 log.go:25] "Validated CRI v1 image API"
Jul 23 21:51:44.923305 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.923285    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 21:51:44.927398 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.927371    2579 fs.go:135] Filesystem UUIDs: map[19bc662f-b389-4685-a9d2-5a589d512348:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f17d47a1-6f15-45f1-b910-3ed391c041e9:/dev/nvme0n1p4]
Jul 23 21:51:44.927494 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.927394    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 21:51:44.931383 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.931349    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 21:51:44.933978 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.933831    2579 manager.go:216] Machine: {Timestamp:2026-07-23 21:51:44.93162105 +0000 UTC m=+0.322013658 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3078561 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eea8b918072b18b0d750eb0e3d215 SystemUUID:ec2eea8b-9180-72b1-8b0d-750eb0e3d215 BootID:f9788160-95b7-46b5-ac71-a81707c128b4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0b:c3:42:fd:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0b:c3:42:fd:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:aa:8d:6f:0e:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 21:51:44.933978 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.933968    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 21:51:44.934136 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.934099    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 21:51:44.937221 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.937188    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 21:51:44.937405 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.937227    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-91.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 21:51:44.937957 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.937945    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 21:51:44.937992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.937960    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 21:51:44.937992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.937971    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 21:51:44.938776 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.938761    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 21:51:44.940426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.940413    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 21:51:44.944096 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.944081    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 21:51:44.944240 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.944230    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 21:51:44.944280 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.944261    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 21:51:44.944280 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.944272    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 21:51:44.944342 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.944283    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 21:51:44.944342 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.944294    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 23 21:51:44.944342 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.944303    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 21:51:44.944840 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.944824    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 21:51:44.944904 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.944854    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 21:51:44.948651 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.948627    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 21:51:44.950953 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.950939    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 21:51:44.951019 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.950971    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 21:51:44.952342 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952329    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 21:51:44.952414 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952347    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 21:51:44.952414 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952353    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 21:51:44.952414 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952358    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 21:51:44.952414 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952367    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 21:51:44.952414 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952373    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 21:51:44.952414 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952381    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 21:51:44.952414 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952386    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 21:51:44.952414 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952393    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 21:51:44.952414 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952398    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 21:51:44.952414 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952406    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 21:51:44.952414 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.952411    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 21:51:44.953157 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.953148    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 21:51:44.953157 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.953158    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 21:51:44.955938 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.955917    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-91.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 21:51:44.956019 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:44.955962    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 21:51:44.956019 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:44.956002    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 21:51:44.956697 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.956685    2579 server.go:1268] "Started kubelet"
Jul 23 21:51:44.956796 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.956763    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 21:51:44.956884 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.956773    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 21:51:44.956884 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.956837    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 21:51:44.957778 ip-10-0-135-91 systemd[1]: Started Kubernetes Kubelet.
Jul 23 21:51:44.958887 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.958853    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 21:51:44.958974 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.958941    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 21:51:44.961069 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:44.961045    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 21:51:44.962668 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.962650    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4lwgs"
Jul 23 21:51:44.965883 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:44.964939    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-91.ec2.internal.18c509b6149acd3f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-91.ec2.internal,UID:ip-10-0-135-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-91.ec2.internal,},FirstTimestamp:2026-07-23 21:51:44.956656959 +0000 UTC m=+0.347049566,LastTimestamp:2026-07-23 21:51:44.956656959 +0000 UTC m=+0.347049566,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-91.ec2.internal,}"
Jul 23 21:51:44.967344 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.967321    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 21:51:44.967344 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.967334    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 21:51:44.967500 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:44.967400    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:44.968066 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.968014    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 21:51:44.968066 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.968033    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 21:51:44.968066 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.968060    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 21:51:44.968292 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.968192    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 21:51:44.968292 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.968204    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 21:51:44.968591 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.968569    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 21:51:44.968591 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.968587    2579 factory.go:55] Registering systemd factory
Jul 23 21:51:44.968591 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.968595    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 23 21:51:44.969972 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.969956    2579 factory.go:153] Registering CRI-O factory
Jul 23 21:51:44.969972 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.969975    2579 factory.go:223] Registration of the crio container factory successfully
Jul 23 21:51:44.970123 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.969999    2579 factory.go:103] Registering Raw factory
Jul 23 21:51:44.970123 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.970012    2579 manager.go:1196] Started watching for new ooms in manager
Jul 23 21:51:44.970418 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:44.970394    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 21:51:44.970513 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:44.970490    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 21:51:44.970660 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.970646    2579 manager.go:349] Starting recovery of all containers
Jul 23 21:51:44.973091 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.973068    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4lwgs"
Jul 23 21:51:44.981609 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.981439    2579 manager.go:354] Recovery completed
Jul 23 21:51:44.986098 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.986085    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 21:51:44.988221 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.988199    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 21:51:44.988307 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.988233    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 21:51:44.988307 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.988244    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 21:51:44.988813 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.988795    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 21:51:44.988813 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.988810    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 21:51:44.988925 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.988827    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 21:51:44.988925 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.988831    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 21:51:44.990087 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.990074    2579 policy_none.go:49] "None policy: Start"
Jul 23 21:51:44.990145 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.990090    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 21:51:44.990145 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.990101    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 21:51:44.990608 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:44.990515    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-91.ec2.internal.18c509b6167c6959  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-91.ec2.internal,UID:ip-10-0-135-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-91.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-91.ec2.internal,},FirstTimestamp:2026-07-23 21:51:44.988219737 +0000 UTC m=+0.378612344,LastTimestamp:2026-07-23 21:51:44.988219737 +0000 UTC m=+0.378612344,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-91.ec2.internal,}"
Jul 23 21:51:44.991025 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.991003    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 21:51:44.991099 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.991043    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 21:51:44.991099 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.991059    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 21:51:44.991441 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:44.991430    2579 policy_none.go:47] "Start"
Jul 23 21:51:45.023748 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.023727    2579 manager.go:337] "Starting Device Plugin manager"
Jul 23 21:51:45.023926 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.023765    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 21:51:45.023926 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.023775    2579 server.go:87] "Starting device plugin registration server"
Jul 23 21:51:45.024063 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.024047    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 21:51:45.024182 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.024065    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 21:51:45.024182 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.024170    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 21:51:45.024289 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.024247    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 21:51:45.024289 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.024253    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 21:51:45.024873 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.024836    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 21:51:45.024957 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.024907    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:45.104221 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.104171    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 21:51:45.105602 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.105585    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 21:51:45.105689 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.105623    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 21:51:45.105689 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.105649    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 21:51:45.105784 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.105681    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 21:51:45.108459 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.108438    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 21:51:45.124883 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.124790    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 21:51:45.125840 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.125824    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 21:51:45.125935 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.125877    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 21:51:45.125935 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.125888    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 21:51:45.125935 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.125913    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.134401 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.134373    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.134469 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.134402    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-91.ec2.internal\": node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:45.152010 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.151986    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:45.206667 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.206621    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"]
Jul 23 21:51:45.206790 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.206715    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 21:51:45.207716 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.207700    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 21:51:45.207792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.207731    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 21:51:45.207792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.207741    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 21:51:45.209076 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.209063    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 21:51:45.209185 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.209171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.209228 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.209219    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 21:51:45.209916 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.209897    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 21:51:45.210009 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.209930    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 21:51:45.210009 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.209940    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 21:51:45.210009 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.209899    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 21:51:45.210009 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.209983    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 21:51:45.210009 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.209998    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 21:51:45.211312 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.211293    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.211409 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.211322    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 21:51:45.212076 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.212063    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 21:51:45.212155 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.212087    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 21:51:45.212155 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.212098    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeHasSufficientPID"
Jul 23 21:51:45.237738 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.237712    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-91.ec2.internal\" not found" node="ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.242382 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.242361    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-91.ec2.internal\" not found" node="ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.252789 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.252760    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:45.270184 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.270157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6784b2a8588e592df3c4dc1aca91e57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"d6784b2a8588e592df3c4dc1aca91e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.270287 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.270188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/599b3085f9eb491ede314f11a46d9b2b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-91.ec2.internal\" (UID: \"599b3085f9eb491ede314f11a46d9b2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.270287 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.270205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6784b2a8588e592df3c4dc1aca91e57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"d6784b2a8588e592df3c4dc1aca91e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.353451 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.353420    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:45.370816 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.370789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6784b2a8588e592df3c4dc1aca91e57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"d6784b2a8588e592df3c4dc1aca91e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.370918 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.370821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6784b2a8588e592df3c4dc1aca91e57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"d6784b2a8588e592df3c4dc1aca91e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.370918 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.370876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/599b3085f9eb491ede314f11a46d9b2b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-91.ec2.internal\" (UID: \"599b3085f9eb491ede314f11a46d9b2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.370990 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.370916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6784b2a8588e592df3c4dc1aca91e57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"d6784b2a8588e592df3c4dc1aca91e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.370990 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.370922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/599b3085f9eb491ede314f11a46d9b2b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-91.ec2.internal\" (UID: \"599b3085f9eb491ede314f11a46d9b2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.370990 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.370924    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6784b2a8588e592df3c4dc1aca91e57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal\" (UID: \"d6784b2a8588e592df3c4dc1aca91e57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.454257 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.454231    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:45.541787 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.541751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.546182 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.545983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:45.555348 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.555325    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:45.655899 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.655838    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:45.756466 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.756397    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:45.777106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.777073    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 21:51:45.858806 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.858755    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:45.898629 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.898585    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 21:51:45.899289 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.898797    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 21:51:45.899289 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.898823    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 21:51:45.959167 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:45.959127    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:45.982371 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.982325    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 21:46:45 +0000 UTC" deadline="2028-02-08 16:09:03.365217863 +0000 UTC"
Jul 23 21:51:45.982371 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:45.982368    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13554h17m17.382854369s"
Jul 23 21:51:46.060141 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:46.060052    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:46.091069 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:46.091029    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6784b2a8588e592df3c4dc1aca91e57.slice/crio-7834f29fcbaefadfa227ec4c3543245ceeaabc2697c95e9b615db07822f42d8a WatchSource:0}: Error finding container 7834f29fcbaefadfa227ec4c3543245ceeaabc2697c95e9b615db07822f42d8a: Status 404 returned error can't find the container with id 7834f29fcbaefadfa227ec4c3543245ceeaabc2697c95e9b615db07822f42d8a
Jul 23 21:51:46.091523 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:46.091497    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599b3085f9eb491ede314f11a46d9b2b.slice/crio-bddec64068529bcb0bd641ef3497a85ad95d5d0a8f732a1a19ddd38cf75814ef WatchSource:0}: Error finding container bddec64068529bcb0bd641ef3497a85ad95d5d0a8f732a1a19ddd38cf75814ef: Status 404 returned error can't find the container with id bddec64068529bcb0bd641ef3497a85ad95d5d0a8f732a1a19ddd38cf75814ef
Jul 23 21:51:46.099844 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.099828    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 21:51:46.109237 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.109161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal" event={"ID":"d6784b2a8588e592df3c4dc1aca91e57","Type":"ContainerStarted","Data":"7834f29fcbaefadfa227ec4c3543245ceeaabc2697c95e9b615db07822f42d8a"}
Jul 23 21:51:46.110281 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.110261    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 21:51:46.110281 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.110273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal" event={"ID":"599b3085f9eb491ede314f11a46d9b2b","Type":"ContainerStarted","Data":"bddec64068529bcb0bd641ef3497a85ad95d5d0a8f732a1a19ddd38cf75814ef"}
Jul 23 21:51:46.160708 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:46.160670    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-91.ec2.internal\" not found"
Jul 23 21:51:46.162234 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.162218    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 21:51:46.167726 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.167706    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:46.180287 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.180263    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 21:51:46.181343 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.181327    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"
Jul 23 21:51:46.194187 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.194166    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 21:51:46.946492 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.946450    2579 apiserver.go:52] "Watching apiserver"
Jul 23 21:51:46.955148 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.955111    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 21:51:46.955612 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.955569    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-n9jpn","openshift-multus/network-metrics-daemon-szssk","openshift-network-diagnostics/network-check-target-zk875","kube-system/konnectivity-agent-m97rx","kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk","openshift-cluster-node-tuning-operator/tuned-vfrqt","openshift-multus/multus-additional-cni-plugins-khkxf","openshift-network-operator/iptables-alerter-s4hp5","openshift-ovn-kubernetes/ovnkube-node-rs2g2","kube-system/global-pull-secret-syncer-455s5","openshift-dns/node-resolver-j8z9l","openshift-image-registry/node-ca-jtp7k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal"]
Jul 23 21:51:46.957634 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.957606    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.959973 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.959951    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c8tnz\""
Jul 23 21:51:46.960194 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.960174    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 21:51:46.960284 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.960271    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 21:51:46.961278 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.961253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:46.961379 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:46.961346    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:51:46.962955 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.962899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:46.963061 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:46.962990    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:51:46.964857 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.964479    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:51:46.965668 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.965575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:46.967054 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.967031    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mg6mh\""
Jul 23 21:51:46.967169 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.967067    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 21:51:46.967237 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.967067    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 21:51:46.967292 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.967236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:46.967563 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.967541    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 21:51:46.968298 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.968271    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6g78c\""
Jul 23 21:51:46.968407 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.968360    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 21:51:46.968606 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.968591    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 21:51:46.968691 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.968605    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 21:51:46.968691 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.968647    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 21:51:46.968799 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.968762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s4hp5"
Jul 23 21:51:46.970155 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.970101    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 21:51:46.970257 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.970152    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 21:51:46.970257 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.970245    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 21:51:46.970401 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.970324    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 21:51:46.970401 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.970354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gf6qx\""
Jul 23 21:51:46.970401 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.970398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jtp7k"
Jul 23 21:51:46.972030 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.972004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:46.972146 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:46.972075    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:51:46.972876 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.972855    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 21:51:46.973036 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.972900    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ds2rt\""
Jul 23 21:51:46.973212 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.973189    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 21:51:46.973328 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.973222    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8dfmj\""
Jul 23 21:51:46.973491 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.973475    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 21:51:46.973750 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.973731    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 21:51:46.973986 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.973965    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 21:51:46.974175 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.974158    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 21:51:46.974752 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.974490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:46.975542 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.975511    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:46.977363 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.976910    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 21:51:46.977363 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.977043    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j8z9l"
Jul 23 21:51:46.977645 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.977627    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 21:51:46.978189 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.978161    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 21:51:46.978280 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.978170    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4nl8p\""
Jul 23 21:51:46.980701 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.980675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-sysctl-d\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.980877 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.980857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-systemd\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.981041 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-var-lib-kubelet\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.981183 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41e05d6b-dfc0-41b4-ad32-474ceb32db15-host-slash\") pod \"iptables-alerter-s4hp5\" (UID: \"41e05d6b-dfc0-41b4-ad32-474ceb32db15\") " pod="openshift-network-operator/iptables-alerter-s4hp5"
Jul 23 21:51:46.981326 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-kubelet-config\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:46.981428 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9eb7954-e1c1-4140-a634-51cb814450cd-agent-certs\") pod \"konnectivity-agent-m97rx\" (UID: \"f9eb7954-e1c1-4140-a634-51cb814450cd\") " pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:51:46.981548 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-os-release\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:46.981548 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh8wt\" (UniqueName: \"kubernetes.io/projected/6eca0599-f929-4362-aedc-edf38f96e1d1-kube-api-access-qh8wt\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:46.981548 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-registration-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:46.981697 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-kubernetes\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.981751 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-sysctl-conf\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.981751 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3130c63-a976-43fd-8e5f-5125c0a090e2-tmp\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.981849 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:46.981938 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:46.981938 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-modprobe-d\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.982045 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-sys\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.982045 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.981999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsw6x\" (UniqueName: \"kubernetes.io/projected/41e05d6b-dfc0-41b4-ad32-474ceb32db15-kube-api-access-nsw6x\") pod \"iptables-alerter-s4hp5\" (UID: \"41e05d6b-dfc0-41b4-ad32-474ceb32db15\") " pod="openshift-network-operator/iptables-alerter-s4hp5"
Jul 23 21:51:46.982145 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.982122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-cnibin\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:46.982196 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.982166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6eca0599-f929-4362-aedc-edf38f96e1d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:46.982242 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.982199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-lib-modules\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.982242 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.982233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtdqc\" (UniqueName: \"kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc\") pod \"network-check-target-zk875\" (UID: \"dfa41a7c-b457-473c-a1cf-5d9eac128eea\") " pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:46.982335 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.982266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9eb7954-e1c1-4140-a634-51cb814450cd-konnectivity-ca\") pod \"konnectivity-agent-m97rx\" (UID: \"f9eb7954-e1c1-4140-a634-51cb814450cd\") " pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:51:46.982386 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.982342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:46.983016 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.982990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-host\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.983128 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-device-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:46.983128 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbn6b\" (UniqueName: \"kubernetes.io/projected/b3130c63-a976-43fd-8e5f-5125c0a090e2-kube-api-access-zbn6b\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.983236 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41e05d6b-dfc0-41b4-ad32-474ceb32db15-iptables-alerter-script\") pod \"iptables-alerter-s4hp5\" (UID: \"41e05d6b-dfc0-41b4-ad32-474ceb32db15\") " pod="openshift-network-operator/iptables-alerter-s4hp5"
Jul 23 21:51:46.983284 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6eca0599-f929-4362-aedc-edf38f96e1d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:46.983343 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6eca0599-f929-4362-aedc-edf38f96e1d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:46.983399 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-socket-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:46.983448 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-sys-fs\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:46.983499 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmr6l\" (UniqueName: \"kubernetes.io/projected/391bb0da-56ca-401c-b294-d127522a4b3d-kube-api-access-vmr6l\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:46.983499 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-sysconfig\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.983652 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-run\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.983652 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-tuned\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:46.983652 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-dbus\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:46.983652 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:46.983652 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.983623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-system-cni-dir\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:46.985426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.984480    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 21:51:46.985426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.984549    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 21:51:46.985426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.984596    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-27b97\""
Jul 23 21:51:46.985426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.984683    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 21:51:46.988609 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.988576    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8zfks\""
Jul 23 21:51:46.988706 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.988675    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 21:51:46.989407 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.989365    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 21:51:46.989569 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.989552    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 21:51:46.991366 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:46.991343    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 21:51:47.006885 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.006852    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8jcxf"
Jul 23 21:51:47.016264 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.016236    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8jcxf"
Jul 23 21:51:47.069629 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.069594    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 21:51:47.084315 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-run-ovn-kubernetes\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.084513 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-env-overrides\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.084513 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-kubelet-config\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:47.084513 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-os-release\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.084513 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-registration-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.084513 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-kubernetes\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.084513 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-kubelet-config\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:47.084795 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3130c63-a976-43fd-8e5f-5125c0a090e2-tmp\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.084795 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084548    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-cnibin\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.084795 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-registration-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.084795 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.084795 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-kubernetes\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.084795 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-os-release\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.084795 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-modprobe-d\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.084795 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-sys\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.084795 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsw6x\" (UniqueName: \"kubernetes.io/projected/41e05d6b-dfc0-41b4-ad32-474ceb32db15-kube-api-access-nsw6x\") pod \"iptables-alerter-s4hp5\" (UID: \"41e05d6b-dfc0-41b4-ad32-474ceb32db15\") " pod="openshift-network-operator/iptables-alerter-s4hp5"
Jul 23 21:51:47.084795 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbfng\" (UniqueName: \"kubernetes.io/projected/53fb3d87-a41e-448b-b0f5-264a123a6811-kube-api-access-fbfng\") pod \"node-resolver-j8z9l\" (UID: \"53fb3d87-a41e-448b-b0f5-264a123a6811\") " pod="openshift-dns/node-resolver-j8z9l"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-sys\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-cnibin\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6eca0599-f929-4362-aedc-edf38f96e1d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba440199-82c0-4d4a-a4ef-29dc6ce4720a-host\") pod \"node-ca-jtp7k\" (UID: \"ba440199-82c0-4d4a-a4ef-29dc6ce4720a\") " pod="openshift-image-registry/node-ca-jtp7k"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5c7f\" (UniqueName: \"kubernetes.io/projected/f7e81cf8-335f-4794-a193-449e38604f3c-kube-api-access-x5c7f\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.084993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-ovnkube-script-lib\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085008    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-system-cni-dir\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-conf-dir\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-cnibin\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdqc\" (UniqueName: \"kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc\") pod \"network-check-target-zk875\" (UID: \"dfa41a7c-b457-473c-a1cf-5d9eac128eea\") " pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9eb7954-e1c1-4140-a634-51cb814450cd-konnectivity-ca\") pod \"konnectivity-agent-m97rx\" (UID: \"f9eb7954-e1c1-4140-a634-51cb814450cd\") " pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-run-openvswitch\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-hostroot\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-run-multus-certs\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.085282 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-device-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-run-netns\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-run-ovn\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-ovn-node-metrics-cert\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-modprobe-d\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmr6l\" (UniqueName: \"kubernetes.io/projected/391bb0da-56ca-401c-b294-d127522a4b3d-kube-api-access-vmr6l\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-sysconfig\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-run\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-systemd-units\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-log-socket\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-cni-netd\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwdcw\" (UniqueName: \"kubernetes.io/projected/5f47c3f9-ee00-43f7-b343-d26d37c20631-kube-api-access-kwdcw\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6eca0599-f929-4362-aedc-edf38f96e1d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53fb3d87-a41e-448b-b0f5-264a123a6811-hosts-file\") pod \"node-resolver-j8z9l\" (UID: \"53fb3d87-a41e-448b-b0f5-264a123a6811\") " pod="openshift-dns/node-resolver-j8z9l"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-dbus\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-cni-bin\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.085999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-os-release\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-var-lib-kubelet\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53fb3d87-a41e-448b-b0f5-264a123a6811-tmp-dir\") pod \"node-resolver-j8z9l\" (UID: \"53fb3d87-a41e-448b-b0f5-264a123a6811\") " pod="openshift-dns/node-resolver-j8z9l"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-system-cni-dir\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-sysctl-d\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-var-lib-kubelet\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.085975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-run\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41e05d6b-dfc0-41b4-ad32-474ceb32db15-host-slash\") pod \"iptables-alerter-s4hp5\" (UID: \"41e05d6b-dfc0-41b4-ad32-474ceb32db15\") " pod="openshift-network-operator/iptables-alerter-s4hp5"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba440199-82c0-4d4a-a4ef-29dc6ce4720a-serviceca\") pod \"node-ca-jtp7k\" (UID: \"ba440199-82c0-4d4a-a4ef-29dc6ce4720a\") " pod="openshift-image-registry/node-ca-jtp7k"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-var-lib-kubelet\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-sysctl-d\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-var-lib-openvswitch\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-device-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9eb7954-e1c1-4140-a634-51cb814450cd-agent-certs\") pod \"konnectivity-agent-m97rx\" (UID: \"f9eb7954-e1c1-4140-a634-51cb814450cd\") " pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-sysconfig\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh8wt\" (UniqueName: \"kubernetes.io/projected/6eca0599-f929-4362-aedc-edf38f96e1d1-kube-api-access-qh8wt\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9eb7954-e1c1-4140-a634-51cb814450cd-konnectivity-ca\") pod \"konnectivity-agent-m97rx\" (UID: \"f9eb7954-e1c1-4140-a634-51cb814450cd\") " pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:51:47.086611 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-dbus\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6eca0599-f929-4362-aedc-edf38f96e1d1-system-cni-dir\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-sysctl-conf\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-run-netns\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41e05d6b-dfc0-41b4-ad32-474ceb32db15-host-slash\") pod \"iptables-alerter-s4hp5\" (UID: \"41e05d6b-dfc0-41b4-ad32-474ceb32db15\") " pod="openshift-network-operator/iptables-alerter-s4hp5"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-node-log\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086558    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-lib-modules\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.086636    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.086723    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret podName:e38142be-62e8-49b7-af8d-cad1fcb9b7c5 nodeName:}" failed. No retries permitted until 2026-07-23 21:51:47.586699573 +0000 UTC m=+2.977092198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret") pod "global-pull-secret-syncer-455s5" (UID: "e38142be-62e8-49b7-af8d-cad1fcb9b7c5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-sysctl-conf\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.086974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-host\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-kubelet\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-run-systemd\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-ovnkube-config\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.087457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-host\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-cni-dir\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-lib-modules\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-socket-dir-parent\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-run-k8s-cni-cncf-io\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbn6b\" (UniqueName: \"kubernetes.io/projected/b3130c63-a976-43fd-8e5f-5125c0a090e2-kube-api-access-zbn6b\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41e05d6b-dfc0-41b4-ad32-474ceb32db15-iptables-alerter-script\") pod \"iptables-alerter-s4hp5\" (UID: \"41e05d6b-dfc0-41b4-ad32-474ceb32db15\") " pod="openshift-network-operator/iptables-alerter-s4hp5"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-etc-openvswitch\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-daemon-config\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-etc-kubernetes\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6eca0599-f929-4362-aedc-edf38f96e1d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6eca0599-f929-4362-aedc-edf38f96e1d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-socket-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-sys-fs\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-tuned\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.087783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-sys-fs\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-socket-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.088345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6eca0599-f929-4362-aedc-edf38f96e1d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41e05d6b-dfc0-41b4-ad32-474ceb32db15-iptables-alerter-script\") pod \"iptables-alerter-s4hp5\" (UID: \"41e05d6b-dfc0-41b4-ad32-474ceb32db15\") " pod="openshift-network-operator/iptables-alerter-s4hp5"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-slash\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088515    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f47c3f9-ee00-43f7-b343-d26d37c20631-cni-binary-copy\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6eca0599-f929-4362-aedc-edf38f96e1d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cfjw\" (UniqueName: \"kubernetes.io/projected/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-kube-api-access-7cfjw\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/391bb0da-56ca-401c-b294-d127522a4b3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-var-lib-cni-bin\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-var-lib-cni-multus\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-systemd\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhk8\" (UniqueName: \"kubernetes.io/projected/ba440199-82c0-4d4a-a4ef-29dc6ce4720a-kube-api-access-gdhk8\") pod \"node-ca-jtp7k\" (UID: \"ba440199-82c0-4d4a-a4ef-29dc6ce4720a\") " pod="openshift-image-registry/node-ca-jtp7k"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088833    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3130c63-a976-43fd-8e5f-5125c0a090e2-tmp\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.089077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.088836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-systemd\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.090261 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.090230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b3130c63-a976-43fd-8e5f-5125c0a090e2-etc-tuned\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.091878 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.091838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9eb7954-e1c1-4140-a634-51cb814450cd-agent-certs\") pod \"konnectivity-agent-m97rx\" (UID: \"f9eb7954-e1c1-4140-a634-51cb814450cd\") " pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:51:47.096445 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.096408    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 21:51:47.096445 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.096438    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 21:51:47.096638 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.096453    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qtdqc for pod openshift-network-diagnostics/network-check-target-zk875: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:51:47.096638 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.096534    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc podName:dfa41a7c-b457-473c-a1cf-5d9eac128eea nodeName:}" failed. No retries permitted until 2026-07-23 21:51:47.596513725 +0000 UTC m=+2.986906342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qtdqc" (UniqueName: "kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc") pod "network-check-target-zk875" (UID: "dfa41a7c-b457-473c-a1cf-5d9eac128eea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:51:47.098783 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.098754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmr6l\" (UniqueName: \"kubernetes.io/projected/391bb0da-56ca-401c-b294-d127522a4b3d-kube-api-access-vmr6l\") pod \"aws-ebs-csi-driver-node-h6sjk\" (UID: \"391bb0da-56ca-401c-b294-d127522a4b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.100114 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.100082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsw6x\" (UniqueName: \"kubernetes.io/projected/41e05d6b-dfc0-41b4-ad32-474ceb32db15-kube-api-access-nsw6x\") pod \"iptables-alerter-s4hp5\" (UID: \"41e05d6b-dfc0-41b4-ad32-474ceb32db15\") " pod="openshift-network-operator/iptables-alerter-s4hp5"
Jul 23 21:51:47.101251 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.101226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbn6b\" (UniqueName: \"kubernetes.io/projected/b3130c63-a976-43fd-8e5f-5125c0a090e2-kube-api-access-zbn6b\") pod \"tuned-vfrqt\" (UID: \"b3130c63-a976-43fd-8e5f-5125c0a090e2\") " pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.101563 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.101539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh8wt\" (UniqueName: \"kubernetes.io/projected/6eca0599-f929-4362-aedc-edf38f96e1d1-kube-api-access-qh8wt\") pod \"multus-additional-cni-plugins-khkxf\" (UID: \"6eca0599-f929-4362-aedc-edf38f96e1d1\") " pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.125527 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.125485    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 21:51:47.189736 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.189692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbfng\" (UniqueName: \"kubernetes.io/projected/53fb3d87-a41e-448b-b0f5-264a123a6811-kube-api-access-fbfng\") pod \"node-resolver-j8z9l\" (UID: \"53fb3d87-a41e-448b-b0f5-264a123a6811\") " pod="openshift-dns/node-resolver-j8z9l"
Jul 23 21:51:47.189967 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.189749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba440199-82c0-4d4a-a4ef-29dc6ce4720a-host\") pod \"node-ca-jtp7k\" (UID: \"ba440199-82c0-4d4a-a4ef-29dc6ce4720a\") " pod="openshift-image-registry/node-ca-jtp7k"
Jul 23 21:51:47.189967 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.189780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5c7f\" (UniqueName: \"kubernetes.io/projected/f7e81cf8-335f-4794-a193-449e38604f3c-kube-api-access-x5c7f\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:47.189967 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.189854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba440199-82c0-4d4a-a4ef-29dc6ce4720a-host\") pod \"node-ca-jtp7k\" (UID: \"ba440199-82c0-4d4a-a4ef-29dc6ce4720a\") " pod="openshift-image-registry/node-ca-jtp7k"
Jul 23 21:51:47.189967 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.189897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-ovnkube-script-lib\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.189967 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.189937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-system-cni-dir\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.189967 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.189954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-conf-dir\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.189991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-run-openvswitch\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-hostroot\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-run-multus-certs\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-conf-dir\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-system-cni-dir\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-run-netns\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190096    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-run-openvswitch\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-run-multus-certs\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-run-ovn\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-hostroot\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-ovn-node-metrics-cert\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-run-netns\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-run-ovn\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-systemd-units\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-log-socket\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-systemd-units\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-cni-netd\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.190309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190284    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-log-socket\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwdcw\" (UniqueName: \"kubernetes.io/projected/5f47c3f9-ee00-43f7-b343-d26d37c20631-kube-api-access-kwdcw\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-cni-netd\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53fb3d87-a41e-448b-b0f5-264a123a6811-hosts-file\") pod \"node-resolver-j8z9l\" (UID: \"53fb3d87-a41e-448b-b0f5-264a123a6811\") " pod="openshift-dns/node-resolver-j8z9l"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-cni-bin\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-os-release\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190410    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/53fb3d87-a41e-448b-b0f5-264a123a6811-hosts-file\") pod \"node-resolver-j8z9l\" (UID: \"53fb3d87-a41e-448b-b0f5-264a123a6811\") " pod="openshift-dns/node-resolver-j8z9l"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-var-lib-kubelet\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53fb3d87-a41e-448b-b0f5-264a123a6811-tmp-dir\") pod \"node-resolver-j8z9l\" (UID: \"53fb3d87-a41e-448b-b0f5-264a123a6811\") " pod="openshift-dns/node-resolver-j8z9l"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba440199-82c0-4d4a-a4ef-29dc6ce4720a-serviceca\") pod \"node-ca-jtp7k\" (UID: \"ba440199-82c0-4d4a-a4ef-29dc6ce4720a\") " pod="openshift-image-registry/node-ca-jtp7k"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-var-lib-openvswitch\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-cni-bin\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-run-netns\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-ovnkube-script-lib\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-run-netns\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-node-log\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191220 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-var-lib-openvswitch\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-kubelet\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-run-systemd\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-node-log\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-ovnkube-config\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-cni-dir\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-socket-dir-parent\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-run-k8s-cni-cncf-io\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-etc-openvswitch\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53fb3d87-a41e-448b-b0f5-264a123a6811-tmp-dir\") pod \"node-resolver-j8z9l\" (UID: \"53fb3d87-a41e-448b-b0f5-264a123a6811\") " pod="openshift-dns/node-resolver-j8z9l"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-daemon-config\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-kubelet\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-os-release\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-run-systemd\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-etc-kubernetes\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-var-lib-kubelet\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-slash\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.191992 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.190963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f47c3f9-ee00-43f7-b343-d26d37c20631-cni-binary-copy\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-etc-kubernetes\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfjw\" (UniqueName: \"kubernetes.io/projected/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-kube-api-access-7cfjw\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-var-lib-cni-bin\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-slash\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-var-lib-cni-multus\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhk8\" (UniqueName: \"kubernetes.io/projected/ba440199-82c0-4d4a-a4ef-29dc6ce4720a-kube-api-access-gdhk8\") pod \"node-ca-jtp7k\" (UID: \"ba440199-82c0-4d4a-a4ef-29dc6ce4720a\") " pod="openshift-image-registry/node-ca-jtp7k"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-run-ovn-kubernetes\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191163    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-var-lib-cni-multus\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-env-overrides\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.191118    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-ovnkube-config\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-cnibin\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-var-lib-cni-bin\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.191232    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs podName:f7e81cf8-335f-4794-a193-449e38604f3c nodeName:}" failed. No retries permitted until 2026-07-23 21:51:47.691212248 +0000 UTC m=+3.081604847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs") pod "network-metrics-daemon-szssk" (UID: "f7e81cf8-335f-4794-a193-449e38604f3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191272    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba440199-82c0-4d4a-a4ef-29dc6ce4720a-serviceca\") pod \"node-ca-jtp7k\" (UID: \"ba440199-82c0-4d4a-a4ef-29dc6ce4720a\") " pod="openshift-image-registry/node-ca-jtp7k"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-host-run-ovn-kubernetes\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-cni-dir\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.192511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-cnibin\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.193234 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-daemon-config\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.193234 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-host-run-k8s-cni-cncf-io\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.193234 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5f47c3f9-ee00-43f7-b343-d26d37c20631-multus-socket-dir-parent\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.193234 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-etc-openvswitch\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.193234 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.191569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f47c3f9-ee00-43f7-b343-d26d37c20631-cni-binary-copy\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.193234 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.192471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-env-overrides\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.193234 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.192955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-ovn-node-metrics-cert\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.200955 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.200923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbfng\" (UniqueName: \"kubernetes.io/projected/53fb3d87-a41e-448b-b0f5-264a123a6811-kube-api-access-fbfng\") pod \"node-resolver-j8z9l\" (UID: \"53fb3d87-a41e-448b-b0f5-264a123a6811\") " pod="openshift-dns/node-resolver-j8z9l"
Jul 23 21:51:47.207982 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.207951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhk8\" (UniqueName: \"kubernetes.io/projected/ba440199-82c0-4d4a-a4ef-29dc6ce4720a-kube-api-access-gdhk8\") pod \"node-ca-jtp7k\" (UID: \"ba440199-82c0-4d4a-a4ef-29dc6ce4720a\") " pod="openshift-image-registry/node-ca-jtp7k"
Jul 23 21:51:47.208121 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.207951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cfjw\" (UniqueName: \"kubernetes.io/projected/295b3136-a93c-4ff3-8e39-cb60ac4a1fa0-kube-api-access-7cfjw\") pod \"ovnkube-node-rs2g2\" (UID: \"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0\") " pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.208286 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.208266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5c7f\" (UniqueName: \"kubernetes.io/projected/f7e81cf8-335f-4794-a193-449e38604f3c-kube-api-access-x5c7f\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:47.209227 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.209198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwdcw\" (UniqueName: \"kubernetes.io/projected/5f47c3f9-ee00-43f7-b343-d26d37c20631-kube-api-access-kwdcw\") pod \"multus-n9jpn\" (UID: \"5f47c3f9-ee00-43f7-b343-d26d37c20631\") " pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.269207 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.269163    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vfrqt"
Jul 23 21:51:47.282118 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.282090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:51:47.300555 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.300520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-khkxf"
Jul 23 21:51:47.305084 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.305052    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk"
Jul 23 21:51:47.315025 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.314992    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s4hp5"
Jul 23 21:51:47.323132 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.323099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jtp7k"
Jul 23 21:51:47.330296 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.330250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:51:47.337553 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.337528    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n9jpn"
Jul 23 21:51:47.343218 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.343189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j8z9l"
Jul 23 21:51:47.595098 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.594998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:47.595280 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.595190    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 21:51:47.595280 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.595263    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret podName:e38142be-62e8-49b7-af8d-cad1fcb9b7c5 nodeName:}" failed. No retries permitted until 2026-07-23 21:51:48.595246058 +0000 UTC m=+3.985638657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret") pod "global-pull-secret-syncer-455s5" (UID: "e38142be-62e8-49b7-af8d-cad1fcb9b7c5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 21:51:47.696120 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.696071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdqc\" (UniqueName: \"kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc\") pod \"network-check-target-zk875\" (UID: \"dfa41a7c-b457-473c-a1cf-5d9eac128eea\") " pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:47.696286 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:47.696179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:47.696286 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.696279    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:51:47.696438 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.696296    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 21:51:47.696438 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.696329    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 21:51:47.696438 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.696344    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qtdqc for pod openshift-network-diagnostics/network-check-target-zk875: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:51:47.696438 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.696351    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs podName:f7e81cf8-335f-4794-a193-449e38604f3c nodeName:}" failed. No retries permitted until 2026-07-23 21:51:48.696329897 +0000 UTC m=+4.086722497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs") pod "network-metrics-daemon-szssk" (UID: "f7e81cf8-335f-4794-a193-449e38604f3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:51:47.696438 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:47.696403    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc podName:dfa41a7c-b457-473c-a1cf-5d9eac128eea nodeName:}" failed. No retries permitted until 2026-07-23 21:51:48.696385244 +0000 UTC m=+4.086777847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qtdqc" (UniqueName: "kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc") pod "network-check-target-zk875" (UID: "dfa41a7c-b457-473c-a1cf-5d9eac128eea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:51:48.024591 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.024298    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 21:46:47 +0000 UTC" deadline="2028-01-28 09:25:43.99020727 +0000 UTC"
Jul 23 21:51:48.024591 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.024331    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13283h33m55.965879181s"
Jul 23 21:51:48.059340 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:48.059302    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba440199_82c0_4d4a_a4ef_29dc6ce4720a.slice/crio-e1f3b2b99135aec0c0b514ba44da2458375ab1f90d76d493f92d2a97b4e09e1b WatchSource:0}: Error finding container e1f3b2b99135aec0c0b514ba44da2458375ab1f90d76d493f92d2a97b4e09e1b: Status 404 returned error can't find the container with id e1f3b2b99135aec0c0b514ba44da2458375ab1f90d76d493f92d2a97b4e09e1b
Jul 23 21:51:48.060477 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:48.060429    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e05d6b_dfc0_41b4_ad32_474ceb32db15.slice/crio-f0ed60ada85d449dc4c43119a920cd69eb3180f54399eb1fe1a3afd51c44384e WatchSource:0}: Error finding container f0ed60ada85d449dc4c43119a920cd69eb3180f54399eb1fe1a3afd51c44384e: Status 404 returned error can't find the container with id f0ed60ada85d449dc4c43119a920cd69eb3180f54399eb1fe1a3afd51c44384e
Jul 23 21:51:48.062648 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:48.062518    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9eb7954_e1c1_4140_a634_51cb814450cd.slice/crio-c95ad90edf0bf8f3a81a0bf40aad6d0065c3621fe965ea518856f429d7e0580e WatchSource:0}: Error finding container c95ad90edf0bf8f3a81a0bf40aad6d0065c3621fe965ea518856f429d7e0580e: Status 404 returned error can't find the container with id c95ad90edf0bf8f3a81a0bf40aad6d0065c3621fe965ea518856f429d7e0580e
Jul 23 21:51:48.063053 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:48.062955    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3130c63_a976_43fd_8e5f_5125c0a090e2.slice/crio-7189bc13fdcc415611cfe1c2722fa43b8e1a86eb529bf014ef6274fad84b1180 WatchSource:0}: Error finding container 7189bc13fdcc415611cfe1c2722fa43b8e1a86eb529bf014ef6274fad84b1180: Status 404 returned error can't find the container with id 7189bc13fdcc415611cfe1c2722fa43b8e1a86eb529bf014ef6274fad84b1180
Jul 23 21:51:48.065357 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:48.064076    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eca0599_f929_4362_aedc_edf38f96e1d1.slice/crio-8ab47d2dd09e3f4a45f0e157f3e6514fb4f0891d9b5755dd75082c7318e0beb1 WatchSource:0}: Error finding container 8ab47d2dd09e3f4a45f0e157f3e6514fb4f0891d9b5755dd75082c7318e0beb1: Status 404 returned error can't find the container with id 8ab47d2dd09e3f4a45f0e157f3e6514fb4f0891d9b5755dd75082c7318e0beb1
Jul 23 21:51:48.065357 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:48.065267    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f47c3f9_ee00_43f7_b343_d26d37c20631.slice/crio-54effab7c3de9381a6031efb7364ed0e9ba890ad01ca1dd29bd3873351e83deb WatchSource:0}: Error finding container 54effab7c3de9381a6031efb7364ed0e9ba890ad01ca1dd29bd3873351e83deb: Status 404 returned error can't find the container with id 54effab7c3de9381a6031efb7364ed0e9ba890ad01ca1dd29bd3873351e83deb
Jul 23 21:51:48.068921 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:48.067358    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295b3136_a93c_4ff3_8e39_cb60ac4a1fa0.slice/crio-4adec88d113e9e5b0a35f54b60c384bfec4bb77284db8e96a0ffa12940b3aa7c WatchSource:0}: Error finding container 4adec88d113e9e5b0a35f54b60c384bfec4bb77284db8e96a0ffa12940b3aa7c: Status 404 returned error can't find the container with id 4adec88d113e9e5b0a35f54b60c384bfec4bb77284db8e96a0ffa12940b3aa7c
Jul 23 21:51:48.073432 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:51:48.073401    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fb3d87_a41e_448b_b0f5_264a123a6811.slice/crio-3372426db2d3036ce6d5d8fd19b57f51387a68b8e5b4508e8e8b48f147060374 WatchSource:0}: Error finding container 3372426db2d3036ce6d5d8fd19b57f51387a68b8e5b4508e8e8b48f147060374: Status 404 returned error can't find the container with id 3372426db2d3036ce6d5d8fd19b57f51387a68b8e5b4508e8e8b48f147060374
Jul 23 21:51:48.119814 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.119592    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2" event={"ID":"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0","Type":"ContainerStarted","Data":"4adec88d113e9e5b0a35f54b60c384bfec4bb77284db8e96a0ffa12940b3aa7c"}
Jul 23 21:51:48.121362 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.121329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khkxf" event={"ID":"6eca0599-f929-4362-aedc-edf38f96e1d1","Type":"ContainerStarted","Data":"8ab47d2dd09e3f4a45f0e157f3e6514fb4f0891d9b5755dd75082c7318e0beb1"}
Jul 23 21:51:48.122549 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.122517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vfrqt" event={"ID":"b3130c63-a976-43fd-8e5f-5125c0a090e2","Type":"ContainerStarted","Data":"7189bc13fdcc415611cfe1c2722fa43b8e1a86eb529bf014ef6274fad84b1180"}
Jul 23 21:51:48.126133 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.126103    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m97rx" event={"ID":"f9eb7954-e1c1-4140-a634-51cb814450cd","Type":"ContainerStarted","Data":"c95ad90edf0bf8f3a81a0bf40aad6d0065c3621fe965ea518856f429d7e0580e"}
Jul 23 21:51:48.131531 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.131501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk" event={"ID":"391bb0da-56ca-401c-b294-d127522a4b3d","Type":"ContainerStarted","Data":"fca29b25a1600c224f721f8fd064f7d773cd2066ad9e21aba0668232cf4ec4ee"}
Jul 23 21:51:48.134850 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.134816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9jpn" event={"ID":"5f47c3f9-ee00-43f7-b343-d26d37c20631","Type":"ContainerStarted","Data":"54effab7c3de9381a6031efb7364ed0e9ba890ad01ca1dd29bd3873351e83deb"}
Jul 23 21:51:48.135984 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.135961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s4hp5" event={"ID":"41e05d6b-dfc0-41b4-ad32-474ceb32db15","Type":"ContainerStarted","Data":"f0ed60ada85d449dc4c43119a920cd69eb3180f54399eb1fe1a3afd51c44384e"}
Jul 23 21:51:48.137158 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.137132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jtp7k" event={"ID":"ba440199-82c0-4d4a-a4ef-29dc6ce4720a","Type":"ContainerStarted","Data":"e1f3b2b99135aec0c0b514ba44da2458375ab1f90d76d493f92d2a97b4e09e1b"}
Jul 23 21:51:48.138255 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.138233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j8z9l" event={"ID":"53fb3d87-a41e-448b-b0f5-264a123a6811","Type":"ContainerStarted","Data":"3372426db2d3036ce6d5d8fd19b57f51387a68b8e5b4508e8e8b48f147060374"}
Jul 23 21:51:48.602382 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.602338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:48.602586 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:48.602568    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 21:51:48.602656 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:48.602644    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret podName:e38142be-62e8-49b7-af8d-cad1fcb9b7c5 nodeName:}" failed. No retries permitted until 2026-07-23 21:51:50.602625389 +0000 UTC m=+5.993017997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret") pod "global-pull-secret-syncer-455s5" (UID: "e38142be-62e8-49b7-af8d-cad1fcb9b7c5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 21:51:48.703476 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.703433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdqc\" (UniqueName: \"kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc\") pod \"network-check-target-zk875\" (UID: \"dfa41a7c-b457-473c-a1cf-5d9eac128eea\") " pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:48.703663 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.703544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:48.703740 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:48.703707    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:51:48.703800 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:48.703780    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs podName:f7e81cf8-335f-4794-a193-449e38604f3c nodeName:}" failed. No retries permitted until 2026-07-23 21:51:50.703758921 +0000 UTC m=+6.094151533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs") pod "network-metrics-daemon-szssk" (UID: "f7e81cf8-335f-4794-a193-449e38604f3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:51:48.704318 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:48.704020    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 21:51:48.704318 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:48.704046    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 21:51:48.704318 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:48.704060    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qtdqc for pod openshift-network-diagnostics/network-check-target-zk875: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:51:48.704318 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:48.704115    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc podName:dfa41a7c-b457-473c-a1cf-5d9eac128eea nodeName:}" failed. No retries permitted until 2026-07-23 21:51:50.704098068 +0000 UTC m=+6.094490686 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qtdqc" (UniqueName: "kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc") pod "network-check-target-zk875" (UID: "dfa41a7c-b457-473c-a1cf-5d9eac128eea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:51:48.757318 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:48.756970    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 21:51:49.025127 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:49.024994    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 21:46:47 +0000 UTC" deadline="2028-05-05 23:38:18.775322086 +0000 UTC"
Jul 23 21:51:49.025127 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:49.025074    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15649h46m29.750253843s"
Jul 23 21:51:49.108697 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:49.107948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:49.108697 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:49.108109    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:51:49.108697 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:49.108548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:49.108697 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:49.108651    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:51:49.109402 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:49.109231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:49.109402 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:49.109354    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:51:49.150461 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:49.150420    2579 generic.go:356] "Generic (PLEG): container finished" podID="d6784b2a8588e592df3c4dc1aca91e57" containerID="bc40f2d4b6423ca98a266686eb97b9749ec79758ad5e306948894878ba40b6be" exitCode=0
Jul 23 21:51:49.150640 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:49.150570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal" event={"ID":"d6784b2a8588e592df3c4dc1aca91e57","Type":"ContainerDied","Data":"bc40f2d4b6423ca98a266686eb97b9749ec79758ad5e306948894878ba40b6be"}
Jul 23 21:51:49.166245 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:49.166205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal" event={"ID":"599b3085f9eb491ede314f11a46d9b2b","Type":"ContainerStarted","Data":"7acb2fd001d422f21d39c6d101a8029b2065678457bfa71a244feb416e2a0150"}
Jul 23 21:51:49.184980 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:49.182257    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-91.ec2.internal" podStartSLOduration=3.182240124 podStartE2EDuration="3.182240124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:51:49.182076299 +0000 UTC m=+4.572468920" watchObservedRunningTime="2026-07-23 21:51:49.182240124 +0000 UTC m=+4.572632743"
Jul 23 21:51:50.181858 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:50.181816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal" event={"ID":"d6784b2a8588e592df3c4dc1aca91e57","Type":"ContainerStarted","Data":"751c3b0c7ee4fd97a35a8f0d450b0b98b8cef128652efd703a5728e311e8c0fc"}
Jul 23 21:51:50.622013 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:50.621964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:50.622230 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:50.622216    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 21:51:50.622305 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:50.622289    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret podName:e38142be-62e8-49b7-af8d-cad1fcb9b7c5 nodeName:}" failed. No retries permitted until 2026-07-23 21:51:54.622269113 +0000 UTC m=+10.012661722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret") pod "global-pull-secret-syncer-455s5" (UID: "e38142be-62e8-49b7-af8d-cad1fcb9b7c5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 21:51:50.723339 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:50.723299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:50.723515 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:50.723368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdqc\" (UniqueName: \"kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc\") pod \"network-check-target-zk875\" (UID: \"dfa41a7c-b457-473c-a1cf-5d9eac128eea\") " pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:50.723602 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:50.723568    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 21:51:50.723602 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:50.723593    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 21:51:50.723725 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:50.723607    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qtdqc for pod openshift-network-diagnostics/network-check-target-zk875: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:51:50.723725 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:50.723671    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc podName:dfa41a7c-b457-473c-a1cf-5d9eac128eea nodeName:}" failed. No retries permitted until 2026-07-23 21:51:54.723650775 +0000 UTC m=+10.114043391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qtdqc" (UniqueName: "kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc") pod "network-check-target-zk875" (UID: "dfa41a7c-b457-473c-a1cf-5d9eac128eea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:51:50.724141 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:50.724120    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:51:50.724232 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:50.724184    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs podName:f7e81cf8-335f-4794-a193-449e38604f3c nodeName:}" failed. No retries permitted until 2026-07-23 21:51:54.72416623 +0000 UTC m=+10.114558829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs") pod "network-metrics-daemon-szssk" (UID: "f7e81cf8-335f-4794-a193-449e38604f3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:51:51.109423 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:51.108506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:51.109423 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:51.108655    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:51:51.109423 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:51.109068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:51.109423 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:51.109171    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:51:51.109423 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:51.109236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:51.109423 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:51.109322    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:51:53.107025 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:53.106989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:53.107498 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:53.106995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:53.107498 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:53.107143    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:51:53.107498 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:53.107302    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:51:53.107498 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:53.107365    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:53.107498 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:53.107447    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:51:54.656953 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:54.656914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:54.657389 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:54.657092    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 21:51:54.657389 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:54.657167    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret podName:e38142be-62e8-49b7-af8d-cad1fcb9b7c5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:02.6571472 +0000 UTC m=+18.047539798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret") pod "global-pull-secret-syncer-455s5" (UID: "e38142be-62e8-49b7-af8d-cad1fcb9b7c5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 21:51:54.758837 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:54.758059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:54.758837 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:54.758121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdqc\" (UniqueName: \"kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc\") pod \"network-check-target-zk875\" (UID: \"dfa41a7c-b457-473c-a1cf-5d9eac128eea\") " pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:54.758837 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:54.758273    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 21:51:54.758837 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:54.758290    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 21:51:54.758837 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:54.758303    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qtdqc for pod openshift-network-diagnostics/network-check-target-zk875: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:51:54.758837 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:54.758365    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc podName:dfa41a7c-b457-473c-a1cf-5d9eac128eea nodeName:}" failed. No retries permitted until 2026-07-23 21:52:02.758342652 +0000 UTC m=+18.148735248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qtdqc" (UniqueName: "kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc") pod "network-check-target-zk875" (UID: "dfa41a7c-b457-473c-a1cf-5d9eac128eea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:51:54.758837 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:54.758752    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:51:54.758837 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:54.758796    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs podName:f7e81cf8-335f-4794-a193-449e38604f3c nodeName:}" failed. No retries permitted until 2026-07-23 21:52:02.758782293 +0000 UTC m=+18.149174894 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs") pod "network-metrics-daemon-szssk" (UID: "f7e81cf8-335f-4794-a193-449e38604f3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:51:55.107377 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:55.107345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:55.107584 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:55.107477    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:51:55.107873 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:55.107848    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:55.107981 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:55.107957    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:51:55.108031 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:55.108024    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:55.108111 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:55.108094    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:51:57.108915 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:57.108171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:57.108915 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:57.108295    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:51:57.108915 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:57.108361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:57.108915 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:57.108446    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:51:57.108915 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:57.108493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:57.108915 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:57.108559    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:51:59.106254 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:59.106213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:51:59.106627 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:59.106252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:51:59.106627 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:51:59.106213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:51:59.106627 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:59.106347    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:51:59.106627 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:59.106390    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:51:59.106627 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:51:59.106470    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:01.106810 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:01.106313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:01.106810 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:01.106318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:01.106810 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:01.106457    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:52:01.106810 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:01.106477    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:01.106810 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:01.106598    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:52:01.106810 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:01.106693    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:02.720462 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:02.720428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:02.720971 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:02.720568    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 21:52:02.720971 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:02.720629    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret podName:e38142be-62e8-49b7-af8d-cad1fcb9b7c5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:18.720614765 +0000 UTC m=+34.111007358 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret") pod "global-pull-secret-syncer-455s5" (UID: "e38142be-62e8-49b7-af8d-cad1fcb9b7c5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 21:52:02.820790 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:02.820754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdqc\" (UniqueName: \"kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc\") pod \"network-check-target-zk875\" (UID: \"dfa41a7c-b457-473c-a1cf-5d9eac128eea\") " pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:02.821006 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:02.820838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:02.821006 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:02.820950    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:52:02.821006 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:02.820980    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 21:52:02.821006 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:02.821005    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 21:52:02.821006 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:02.821008    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs podName:f7e81cf8-335f-4794-a193-449e38604f3c nodeName:}" failed. No retries permitted until 2026-07-23 21:52:18.820987496 +0000 UTC m=+34.211380105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs") pod "network-metrics-daemon-szssk" (UID: "f7e81cf8-335f-4794-a193-449e38604f3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:52:02.821195 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:02.821019    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qtdqc for pod openshift-network-diagnostics/network-check-target-zk875: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:52:02.821195 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:02.821074    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc podName:dfa41a7c-b457-473c-a1cf-5d9eac128eea nodeName:}" failed. No retries permitted until 2026-07-23 21:52:18.821058575 +0000 UTC m=+34.211451174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qtdqc" (UniqueName: "kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc") pod "network-check-target-zk875" (UID: "dfa41a7c-b457-473c-a1cf-5d9eac128eea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:52:03.106725 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:03.106693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:03.106921 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:03.106693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:03.106921 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:03.106823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:03.106921 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:03.106833    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:52:03.107054 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:03.106993    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:52:03.107125 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:03.107098    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:05.107479 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:05.107425    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:05.107928 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:05.107535    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:05.107928 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:05.107550    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:52:05.107928 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:05.107601    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:05.107928 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:05.107690    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:05.107928 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:05.107768    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:52:07.106384 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:07.106348    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:07.106966 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:07.106345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:07.106966 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:07.106492    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:52:07.106966 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:07.106348    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:07.106966 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:07.106574    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:07.106966 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:07.106627    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:52:08.220491 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.220210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk" event={"ID":"391bb0da-56ca-401c-b294-d127522a4b3d","Type":"ContainerStarted","Data":"7c4dc99d72b6f025decde9e926032107d465cdfcad577491c108118d63478505"}
Jul 23 21:52:08.221937 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.221896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n9jpn" event={"ID":"5f47c3f9-ee00-43f7-b343-d26d37c20631","Type":"ContainerStarted","Data":"f7186e8cf7f06d4c2dc0e636692fab8bd7ed11d660e289c7bb3f0920f3b06c26"}
Jul 23 21:52:08.223711 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.223685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jtp7k" event={"ID":"ba440199-82c0-4d4a-a4ef-29dc6ce4720a","Type":"ContainerStarted","Data":"18e306468d97784468e63f9bf7e7d2acfacd8b88ab02e06cbfdb3b2cb7104589"}
Jul 23 21:52:08.225354 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.225332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j8z9l" event={"ID":"53fb3d87-a41e-448b-b0f5-264a123a6811","Type":"ContainerStarted","Data":"172f71da94f0923a3f8ebf1e447a55b0d3259526ccedea46c4e11836dfe43ae1"}
Jul 23 21:52:08.231231 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.231209    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 21:52:08.231622 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.231594    2579 generic.go:356] "Generic (PLEG): container finished" podID="295b3136-a93c-4ff3-8e39-cb60ac4a1fa0" containerID="1ba5f04ad07fc997a27647f998f1f13dddf9b3f64f443f62ab2f290c3f8f81d4" exitCode=1
Jul 23 21:52:08.231723 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.231659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2" event={"ID":"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0","Type":"ContainerStarted","Data":"5dd58530699ab9ec401bca43fb68f7e72da948c6e481feeaf51f1c3a9bd47436"}
Jul 23 21:52:08.231770 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.231752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2" event={"ID":"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0","Type":"ContainerStarted","Data":"a85af99de523e87d356586c345aa7c97590bfdbdb41aa02dd68398f3f001563c"}
Jul 23 21:52:08.231803 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.231784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2" event={"ID":"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0","Type":"ContainerStarted","Data":"656d655b1f1367f78309655f81e0588266745c27d00b84fa72ab8ab4abccea12"}
Jul 23 21:52:08.231832 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.231813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2" event={"ID":"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0","Type":"ContainerDied","Data":"1ba5f04ad07fc997a27647f998f1f13dddf9b3f64f443f62ab2f290c3f8f81d4"}
Jul 23 21:52:08.231896 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.231851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2" event={"ID":"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0","Type":"ContainerStarted","Data":"c705bd7aecb28ce34eb9bd8c62dd6bffcf96efca9a8888b921ffdbf2bde4e5d2"}
Jul 23 21:52:08.233513 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.233483    2579 generic.go:356] "Generic (PLEG): container finished" podID="6eca0599-f929-4362-aedc-edf38f96e1d1" containerID="fbccf280828bbf92c5bbba2688e2e464d42b1947c61403b268f97d3d6551bab0" exitCode=0
Jul 23 21:52:08.233750 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.233728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khkxf" event={"ID":"6eca0599-f929-4362-aedc-edf38f96e1d1","Type":"ContainerDied","Data":"fbccf280828bbf92c5bbba2688e2e464d42b1947c61403b268f97d3d6551bab0"}
Jul 23 21:52:08.235701 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.235675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vfrqt" event={"ID":"b3130c63-a976-43fd-8e5f-5125c0a090e2","Type":"ContainerStarted","Data":"49faa49ffa0cba7e18274d7385dadcdf0e9237a746e2fce8bfa84470e8b4b12a"}
Jul 23 21:52:08.237169 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.237148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m97rx" event={"ID":"f9eb7954-e1c1-4140-a634-51cb814450cd","Type":"ContainerStarted","Data":"4a799688e5d4318de1eab6aa226abcc3b807090fe9f08c3a3239cf320a161ca1"}
Jul 23 21:52:08.239924 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.239888    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-91.ec2.internal" podStartSLOduration=22.239846709 podStartE2EDuration="22.239846709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:51:50.201162862 +0000 UTC m=+5.591555483" watchObservedRunningTime="2026-07-23 21:52:08.239846709 +0000 UTC m=+23.630239326"
Jul 23 21:52:08.240041 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.239964    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n9jpn" podStartSLOduration=3.793078447 podStartE2EDuration="23.23995865s" totalImagesPullingTime="19.446880203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:08.239707027 +0000 UTC m=+23.630099644" watchObservedRunningTime="2026-07-23 21:52:08.23995865 +0000 UTC m=+23.630351265"
Jul 23 21:52:08.255508 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.254974    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jtp7k" podStartSLOduration=3.848465875 podStartE2EDuration="23.254847123s" totalImagesPullingTime="19.406381248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:08.254023881 +0000 UTC m=+23.644416498" watchObservedRunningTime="2026-07-23 21:52:08.254847123 +0000 UTC m=+23.645239743"
Jul 23 21:52:08.269483 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.269420    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j8z9l" podStartSLOduration=3.85344196 podStartE2EDuration="23.269401058s" totalImagesPullingTime="19.415959098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:08.269281806 +0000 UTC m=+23.659674420" watchObservedRunningTime="2026-07-23 21:52:08.269401058 +0000 UTC m=+23.659793677"
Jul 23 21:52:08.296005 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.295946    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vfrqt" podStartSLOduration=3.868397648 podStartE2EDuration="23.295917029s" totalImagesPullingTime="19.427519381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:08.295484601 +0000 UTC m=+23.685877219" watchObservedRunningTime="2026-07-23 21:52:08.295917029 +0000 UTC m=+23.686309648"
Jul 23 21:52:08.319103 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.319057    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m97rx" podStartSLOduration=8.219695536 podStartE2EDuration="23.319042354s" totalImagesPullingTime="15.099346818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:08.318356271 +0000 UTC m=+23.708748889" watchObservedRunningTime="2026-07-23 21:52:08.319042354 +0000 UTC m=+23.709434969"
Jul 23 21:52:08.410458 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.410421    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:08.491043 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:08.490931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:09.020692 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.020664    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 21:52:09.037982 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.037887    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T21:52:09.020684727Z","UUID":"7df48b01-901a-4f3d-a950-f7bfccdc4aed","Handler":null,"Name":"","Endpoint":""}
Jul 23 21:52:09.039550 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.039480    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 21:52:09.039550 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.039527    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 21:52:09.057056 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.057025    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:09.106432 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.106398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:09.106620 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.106398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:09.106620 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:09.106547    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:09.106620 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:09.106594    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:52:09.106620 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.106398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:09.106828 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:09.106687    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:52:09.164456 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.164420    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:09.242152 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.242115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk" event={"ID":"391bb0da-56ca-401c-b294-d127522a4b3d","Type":"ContainerStarted","Data":"3e785dc7e5c8846306505089f9c8bece8f0b1966649a0ae25ab61f507e55d643"}
Jul 23 21:52:09.243873 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.243837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s4hp5" event={"ID":"41e05d6b-dfc0-41b4-ad32-474ceb32db15","Type":"ContainerStarted","Data":"031b966dfb4794278d19cf5b11ec59a27d096d9050e03b1dad20d3d43a2d8e6a"}
Jul 23 21:52:09.247644 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.247620    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 21:52:09.248070 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.248041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2" event={"ID":"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0","Type":"ContainerStarted","Data":"967a6f4f0c600898a8fe028e2246a6f6e8e9f43b7280358eb4812ee0cbbd9f0d"}
Jul 23 21:52:09.265815 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.265761    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s4hp5" podStartSLOduration=4.860340635 podStartE2EDuration="24.265741972s" totalImagesPullingTime="19.405401337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:09.259641526 +0000 UTC m=+24.650034140" watchObservedRunningTime="2026-07-23 21:52:09.265741972 +0000 UTC m=+24.656134592"
Jul 23 21:52:09.379494 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.379404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:09.709971 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.709925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:09.776319 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:09.776280    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:10.253105 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:10.253067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk" event={"ID":"391bb0da-56ca-401c-b294-d127522a4b3d","Type":"ContainerStarted","Data":"1b0189f22fbba2aa2ccfab1463c0719b1f0037abee162ea2237333c12bd3c6ba"}
Jul 23 21:52:10.279689 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:10.279634    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h6sjk" podStartSLOduration=3.614474753 podStartE2EDuration="25.279615086s" totalImagesPullingTime="21.665140333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:10.279107554 +0000 UTC m=+25.669500171" watchObservedRunningTime="2026-07-23 21:52:10.279615086 +0000 UTC m=+25.670007702"
Jul 23 21:52:10.485798 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:10.485590    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:10.582083 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:10.582049    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:11.106360 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:11.106324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:11.106534 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:11.106367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:11.106534 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:11.106332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:11.106534 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:11.106453    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:52:11.106702 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:11.106545    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:52:11.106702 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:11.106660    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:11.193575 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:11.193540    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:52:11.194219 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:11.194199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:52:11.259170 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:11.259142    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 21:52:11.259636 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:11.259559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2" event={"ID":"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0","Type":"ContainerStarted","Data":"e01c618f602fe9a79b46e27e7f560d9bd6519888b997dcb8a5811dc56c19e47d"}
Jul 23 21:52:11.259993 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:11.259957    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:52:11.260391 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:11.260371    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m97rx"
Jul 23 21:52:11.330803 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:11.330775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:12.188838 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:12.188802    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:12.254935 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:12.254880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:12.811193 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:12.810949    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:12.912388 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:12.912347    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:13.106742 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:13.106622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:13.106742 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:13.106626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:13.106996 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:13.106750    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:13.106996 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:13.106771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:13.106996 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:13.106899    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:52:13.107124 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:13.106998    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:52:13.267375 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:13.267349    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 21:52:13.267725 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:13.267685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2" event={"ID":"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0","Type":"ContainerStarted","Data":"ae33697ae8329da31ce9fce5ac0ce022a3598cef56fb12903f623fba7f3e0491"}
Jul 23 21:52:13.267980 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:13.267952    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:52:13.268065 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:13.268036    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:52:13.268163 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:13.268147    2579 scope.go:117] "RemoveContainer" containerID="1ba5f04ad07fc997a27647f998f1f13dddf9b3f64f443f62ab2f290c3f8f81d4"
Jul 23 21:52:13.270115 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:13.270092    2579 generic.go:356] "Generic (PLEG): container finished" podID="6eca0599-f929-4362-aedc-edf38f96e1d1" containerID="d992bbe137819b2647ad12dc642740daa3ed9a7868e1b5fa87b8cf7b09ed00e5" exitCode=0
Jul 23 21:52:13.270205 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:13.270139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khkxf" event={"ID":"6eca0599-f929-4362-aedc-edf38f96e1d1","Type":"ContainerDied","Data":"d992bbe137819b2647ad12dc642740daa3ed9a7868e1b5fa87b8cf7b09ed00e5"}
Jul 23 21:52:13.285820 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:13.285799    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:52:14.277043 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.277017    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 21:52:14.277683 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.277401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2" event={"ID":"295b3136-a93c-4ff3-8e39-cb60ac4a1fa0","Type":"ContainerStarted","Data":"fa6eb2d9dab816ff00f491c63cc74311de82062424dd7f21e9c7009306c55d44"}
Jul 23 21:52:14.277683 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.277575    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:52:14.294676 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.294652    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:52:14.318367 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.318318    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2" podStartSLOduration=9.829216626000001 podStartE2EDuration="29.318303468s" totalImagesPullingTime="19.489086842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:14.318030525 +0000 UTC m=+29.708423142" watchObservedRunningTime="2026-07-23 21:52:14.318303468 +0000 UTC m=+29.708696085"
Jul 23 21:52:14.584261 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.584031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:14.625560 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.625523    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-455s5"]
Jul 23 21:52:14.625718 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.625666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:14.625805 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:14.625781    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:52:14.628568 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.628540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zk875"]
Jul 23 21:52:14.628690 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.628644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:14.628733 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:14.628717    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:52:14.630247 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.630223    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-szssk"]
Jul 23 21:52:14.630370 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.630322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:14.630419 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:14.630400    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:14.685646 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:14.685613    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:15.282219 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:15.282186    2579 generic.go:356] "Generic (PLEG): container finished" podID="6eca0599-f929-4362-aedc-edf38f96e1d1" containerID="9fd835e5ec1ff4157508c947d4af016668e097e276a32afb5e5f91814c3d7bb9" exitCode=0
Jul 23 21:52:15.282768 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:15.282276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khkxf" event={"ID":"6eca0599-f929-4362-aedc-edf38f96e1d1","Type":"ContainerDied","Data":"9fd835e5ec1ff4157508c947d4af016668e097e276a32afb5e5f91814c3d7bb9"}
Jul 23 21:52:16.106505 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:16.106462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:16.106703 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:16.106462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:16.106703 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:16.106589    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:16.106703 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:16.106665    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:52:17.106244 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:17.106192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:17.106969 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:17.106316    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:52:17.289880 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:17.289826    2579 generic.go:356] "Generic (PLEG): container finished" podID="6eca0599-f929-4362-aedc-edf38f96e1d1" containerID="9065c173f6c2403ebb92af90e834b2a5995f4d184568912736e550e3a7ab9c83" exitCode=0
Jul 23 21:52:17.290029 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:17.289918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khkxf" event={"ID":"6eca0599-f929-4362-aedc-edf38f96e1d1","Type":"ContainerDied","Data":"9065c173f6c2403ebb92af90e834b2a5995f4d184568912736e550e3a7ab9c83"}
Jul 23 21:52:18.092733 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.092633    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:18.106466 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.106417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:18.106466 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.106458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:18.107155 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:18.106596    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:52:18.107155 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:18.106688    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:18.191765 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.191725    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:18.344723 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.344634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:18.426627 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.426591    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:18.502459 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.502412    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:18.737199 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.737169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:18.737408 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:18.737320    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 21:52:18.737408 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:18.737382    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret podName:e38142be-62e8-49b7-af8d-cad1fcb9b7c5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:50.737365486 +0000 UTC m=+66.127758081 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret") pod "global-pull-secret-syncer-455s5" (UID: "e38142be-62e8-49b7-af8d-cad1fcb9b7c5") : object "kube-system"/"original-pull-secret" not registered
Jul 23 21:52:18.834496 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.834447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:18.838309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.838276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:18.838462 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.838332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdqc\" (UniqueName: \"kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc\") pod \"network-check-target-zk875\" (UID: \"dfa41a7c-b457-473c-a1cf-5d9eac128eea\") " pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:18.838462 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:18.838433    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:52:18.838561 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:18.838480    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 21:52:18.838561 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:18.838496    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 21:52:18.838561 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:18.838503    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs podName:f7e81cf8-335f-4794-a193-449e38604f3c nodeName:}" failed. No retries permitted until 2026-07-23 21:52:50.838483958 +0000 UTC m=+66.228876554 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs") pod "network-metrics-daemon-szssk" (UID: "f7e81cf8-335f-4794-a193-449e38604f3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 21:52:18.838561 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:18.838537    2579 projected.go:196] Error preparing data for projected volume kube-api-access-qtdqc for pod openshift-network-diagnostics/network-check-target-zk875: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:52:18.838712 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:18.838570    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc podName:dfa41a7c-b457-473c-a1cf-5d9eac128eea nodeName:}" failed. No retries permitted until 2026-07-23 21:52:50.838559748 +0000 UTC m=+66.228952356 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qtdqc" (UniqueName: "kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc") pod "network-check-target-zk875" (UID: "dfa41a7c-b457-473c-a1cf-5d9eac128eea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 21:52:18.981167 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:18.981129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:19.062688 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:19.062642    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:19.105998 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:19.105966    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:19.106179 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:19.106095    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:52:19.338117 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:19.338024    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:19.441165 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:19.441113    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:19.603346 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:19.603240    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:19.700016 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:19.699884    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:19.830696 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:19.830668    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j8z9l_53fb3d87-a41e-448b-b0f5-264a123a6811/dns-node-resolver/0.log"
Jul 23 21:52:20.106820 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:20.106783    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:20.107035 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:20.106829    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:20.107035 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:20.106933    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-szssk" podUID="f7e81cf8-335f-4794-a193-449e38604f3c"
Jul 23 21:52:20.107152 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:20.107073    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zk875" podUID="dfa41a7c-b457-473c-a1cf-5d9eac128eea"
Jul 23 21:52:20.830712 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:20.830674    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jtp7k_ba440199-82c0-4d4a-a4ef-29dc6ce4720a/node-ca/0.log"
Jul 23 21:52:21.106174 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.106088    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:21.106337 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:21.106238    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-455s5" podUID="e38142be-62e8-49b7-af8d-cad1fcb9b7c5"
Jul 23 21:52:21.235434 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.235394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:21.301380 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.301334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:21.453229 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.453206    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-91.ec2.internal" event="NodeReady"
Jul 23 21:52:21.453371 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.453338    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 21:52:21.507995 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.507962    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l5bhg"]
Jul 23 21:52:21.521605 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.521566    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rpp2m"]
Jul 23 21:52:21.521768 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.521715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.524561 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.524527    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 21:52:21.524809 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.524794    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 21:52:21.524896 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.524819    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wq952\""
Jul 23 21:52:21.540009 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.539967    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l5bhg"]
Jul 23 21:52:21.540009 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.540012    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rpp2m"]
Jul 23 21:52:21.540199 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.540036    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5jh4m"]
Jul 23 21:52:21.540199 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.540096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rpp2m"
Jul 23 21:52:21.544202 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.544170    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 21:52:21.544339 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.544272    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-swdkn\""
Jul 23 21:52:21.545563 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.545541    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 21:52:21.547356 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.547336    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 21:52:21.560154 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.560120    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5jh4m"]
Jul 23 21:52:21.560286 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.560241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.563166 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.563142    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 21:52:21.563424 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.563407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jtqd5\""
Jul 23 21:52:21.563476 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.563407    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 21:52:21.563695 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.563672    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 21:52:21.564107 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.564089    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 21:52:21.581681 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.581641    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:21.661563 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.661513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsb94\" (UniqueName: \"kubernetes.io/projected/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-api-access-nsb94\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.661563 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.661564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sr5d\" (UniqueName: \"kubernetes.io/projected/b3fb3e9c-a9ef-4631-b30b-645d9e5e4388-kube-api-access-4sr5d\") pod \"ingress-canary-rpp2m\" (UID: \"b3fb3e9c-a9ef-4631-b30b-645d9e5e4388\") " pod="openshift-ingress-canary/ingress-canary-rpp2m"
Jul 23 21:52:21.661792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.661650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59368801-3baf-4007-bf31-62e00a4a8306-config-volume\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.661792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.661678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff5h4\" (UniqueName: \"kubernetes.io/projected/59368801-3baf-4007-bf31-62e00a4a8306-kube-api-access-ff5h4\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.661792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.661719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3fb3e9c-a9ef-4631-b30b-645d9e5e4388-cert\") pod \"ingress-canary-rpp2m\" (UID: \"b3fb3e9c-a9ef-4631-b30b-645d9e5e4388\") " pod="openshift-ingress-canary/ingress-canary-rpp2m"
Jul 23 21:52:21.661792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.661736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59368801-3baf-4007-bf31-62e00a4a8306-tmp-dir\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.661792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.661762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-crio-socket\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.662057 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.661802    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.662057 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.661906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.662057 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.661943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-data-volume\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.662057 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.661979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.676271 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.676229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:21.745421 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.745377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:21.763329 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsb94\" (UniqueName: \"kubernetes.io/projected/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-api-access-nsb94\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.763432 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sr5d\" (UniqueName: \"kubernetes.io/projected/b3fb3e9c-a9ef-4631-b30b-645d9e5e4388-kube-api-access-4sr5d\") pod \"ingress-canary-rpp2m\" (UID: \"b3fb3e9c-a9ef-4631-b30b-645d9e5e4388\") " pod="openshift-ingress-canary/ingress-canary-rpp2m"
Jul 23 21:52:21.763432 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59368801-3baf-4007-bf31-62e00a4a8306-config-volume\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.763432 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff5h4\" (UniqueName: \"kubernetes.io/projected/59368801-3baf-4007-bf31-62e00a4a8306-kube-api-access-ff5h4\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.763550 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3fb3e9c-a9ef-4631-b30b-645d9e5e4388-cert\") pod \"ingress-canary-rpp2m\" (UID: \"b3fb3e9c-a9ef-4631-b30b-645d9e5e4388\") " pod="openshift-ingress-canary/ingress-canary-rpp2m"
Jul 23 21:52:21.763550 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59368801-3baf-4007-bf31-62e00a4a8306-tmp-dir\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.763550 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-crio-socket\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.763550 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.763550 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.763789 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-data-volume\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.763789 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.763789 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:21.763677    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 21:52:21.763789 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:21.763706    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 21:52:21.763789 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:21.763716    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 21:52:21.763789 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:21.763768    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls podName:2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:22.263744276 +0000 UTC m=+37.654136873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jh4m" (UID: "2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5") : secret "insights-runtime-extractor-tls" not found
Jul 23 21:52:21.763789 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:21.763794    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls podName:59368801-3baf-4007-bf31-62e00a4a8306 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:22.26378094 +0000 UTC m=+37.654173537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls") pod "dns-default-l5bhg" (UID: "59368801-3baf-4007-bf31-62e00a4a8306") : secret "dns-default-metrics-tls" not found
Jul 23 21:52:21.764157 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.763805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-crio-socket\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.764157 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:21.763811    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm podName:2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:22.263802082 +0000 UTC m=+37.654194680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jh4m" (UID: "2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5") : configmap "kube-rbac-proxy" not found
Jul 23 21:52:21.764157 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.764022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59368801-3baf-4007-bf31-62e00a4a8306-tmp-dir\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.764157 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.764097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/59368801-3baf-4007-bf31-62e00a4a8306-config-volume\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.764157 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.764105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-data-volume\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.768280 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.768255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3fb3e9c-a9ef-4631-b30b-645d9e5e4388-cert\") pod \"ingress-canary-rpp2m\" (UID: \"b3fb3e9c-a9ef-4631-b30b-645d9e5e4388\") " pod="openshift-ingress-canary/ingress-canary-rpp2m"
Jul 23 21:52:21.773954 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.773929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff5h4\" (UniqueName: \"kubernetes.io/projected/59368801-3baf-4007-bf31-62e00a4a8306-kube-api-access-ff5h4\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:21.774072 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.774021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sr5d\" (UniqueName: \"kubernetes.io/projected/b3fb3e9c-a9ef-4631-b30b-645d9e5e4388-kube-api-access-4sr5d\") pod \"ingress-canary-rpp2m\" (UID: \"b3fb3e9c-a9ef-4631-b30b-645d9e5e4388\") " pod="openshift-ingress-canary/ingress-canary-rpp2m"
Jul 23 21:52:21.774255 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.774238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsb94\" (UniqueName: \"kubernetes.io/projected/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-api-access-nsb94\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:21.852977 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.852948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rpp2m"
Jul 23 21:52:21.995115 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:21.995000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rpp2m"]
Jul 23 21:52:21.998826 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:52:21.998789    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3fb3e9c_a9ef_4631_b30b_645d9e5e4388.slice/crio-cca707379ae2f5bc9e95207c86bdb3a1f4439de18adcc68741df06a536556fd1 WatchSource:0}: Error finding container cca707379ae2f5bc9e95207c86bdb3a1f4439de18adcc68741df06a536556fd1: Status 404 returned error can't find the container with id cca707379ae2f5bc9e95207c86bdb3a1f4439de18adcc68741df06a536556fd1
Jul 23 21:52:22.105980 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.105914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:22.105980 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.105958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:22.109280 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.109251    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 21:52:22.109428 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.109397    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 21:52:22.110258 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.110238    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fxtft\""
Jul 23 21:52:22.110375 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.110261    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 21:52:22.110375 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.110306    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n9zwh\""
Jul 23 21:52:22.267216 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.267122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:22.267216 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.267160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:22.267216 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.267183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:22.267466 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:22.267271    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 21:52:22.267466 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:22.267283    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 21:52:22.267466 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:22.267285    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 21:52:22.267466 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:22.267337    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls podName:59368801-3baf-4007-bf31-62e00a4a8306 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:23.267321509 +0000 UTC m=+38.657714103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls") pod "dns-default-l5bhg" (UID: "59368801-3baf-4007-bf31-62e00a4a8306") : secret "dns-default-metrics-tls" not found
Jul 23 21:52:22.267466 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:22.267372    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm podName:2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:23.267349585 +0000 UTC m=+38.657742178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jh4m" (UID: "2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5") : configmap "kube-rbac-proxy" not found
Jul 23 21:52:22.267466 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:22.267385    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls podName:2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:23.267379884 +0000 UTC m=+38.657772478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jh4m" (UID: "2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5") : secret "insights-runtime-extractor-tls" not found
Jul 23 21:52:22.303670 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.303627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rpp2m" event={"ID":"b3fb3e9c-a9ef-4631-b30b-645d9e5e4388","Type":"ContainerStarted","Data":"cca707379ae2f5bc9e95207c86bdb3a1f4439de18adcc68741df06a536556fd1"}
Jul 23 21:52:22.306335 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.306306    2579 generic.go:356] "Generic (PLEG): container finished" podID="6eca0599-f929-4362-aedc-edf38f96e1d1" containerID="a2d518eb273ebe40789c24924b7ef18b93bee5de467500857873e3a58c262198" exitCode=0
Jul 23 21:52:22.306483 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.306348    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khkxf" event={"ID":"6eca0599-f929-4362-aedc-edf38f96e1d1","Type":"ContainerDied","Data":"a2d518eb273ebe40789c24924b7ef18b93bee5de467500857873e3a58c262198"}
Jul 23 21:52:22.351795 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:22.351752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:23.106616 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:23.106580    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:23.109132 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:23.109104    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 21:52:23.275593 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:23.275398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:23.275750 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:23.275618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:23.275750 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:23.275645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:23.275951 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:23.275566    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 21:52:23.275951 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:23.275849    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls podName:2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:25.275835573 +0000 UTC m=+40.666228167 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jh4m" (UID: "2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5") : secret "insights-runtime-extractor-tls" not found
Jul 23 21:52:23.275951 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:23.275940    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 21:52:23.276114 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:23.275965    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls podName:59368801-3baf-4007-bf31-62e00a4a8306 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:25.275957387 +0000 UTC m=+40.666349981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls") pod "dns-default-l5bhg" (UID: "59368801-3baf-4007-bf31-62e00a4a8306") : secret "dns-default-metrics-tls" not found
Jul 23 21:52:23.276114 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:23.275795    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 21:52:23.276114 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:23.275991    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm podName:2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:25.275985456 +0000 UTC m=+40.666378049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jh4m" (UID: "2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5") : configmap "kube-rbac-proxy" not found
Jul 23 21:52:23.312809 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:23.312780    2579 generic.go:356] "Generic (PLEG): container finished" podID="6eca0599-f929-4362-aedc-edf38f96e1d1" containerID="562e8022275ef74809c1f2c34948a732c41359ec6a7249919bb15bdca45de672" exitCode=0
Jul 23 21:52:23.312962 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:23.312838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khkxf" event={"ID":"6eca0599-f929-4362-aedc-edf38f96e1d1","Type":"ContainerDied","Data":"562e8022275ef74809c1f2c34948a732c41359ec6a7249919bb15bdca45de672"}
Jul 23 21:52:24.316449 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:24.316363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rpp2m" event={"ID":"b3fb3e9c-a9ef-4631-b30b-645d9e5e4388","Type":"ContainerStarted","Data":"743deeafa4b24bef77005a60916856ee2f2d35ba5570cba6b0a64f61a787ef06"}
Jul 23 21:52:24.319737 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:24.319697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-khkxf" event={"ID":"6eca0599-f929-4362-aedc-edf38f96e1d1","Type":"ContainerStarted","Data":"b0abdfb1d6c6109f41c25e35c7b59107c78e877cc1cb4826e10ce2feeb6ca1c7"}
Jul 23 21:52:24.335154 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:24.335106    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rpp2m" podStartSLOduration=1.284263974 podStartE2EDuration="3.335089387s" totalImagesPullingTime="2.050825413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:24.334376728 +0000 UTC m=+39.724769345" watchObservedRunningTime="2026-07-23 21:52:24.335089387 +0000 UTC m=+39.725482003"
Jul 23 21:52:24.358299 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:24.358191    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-khkxf" podStartSLOduration=9.17035086 podStartE2EDuration="39.358169167s" totalImagesPullingTime="30.187818307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:24.356909815 +0000 UTC m=+39.747302431" watchObservedRunningTime="2026-07-23 21:52:24.358169167 +0000 UTC m=+39.748561783"
Jul 23 21:52:24.735679 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:24.735645    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:24.837388 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:24.837351    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:24.937953 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:24.937914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:25.039149 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:25.039053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:25.096975 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:25.096945    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:25.274723 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:25.274686    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:25.288733 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:25.288696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:25.288917 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:25.288746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:25.288917 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:25.288852    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 21:52:25.289015 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:25.288928    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 21:52:25.289015 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:25.288850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:25.289015 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:25.288938    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls podName:59368801-3baf-4007-bf31-62e00a4a8306 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:29.288920891 +0000 UTC m=+44.679313485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls") pod "dns-default-l5bhg" (UID: "59368801-3baf-4007-bf31-62e00a4a8306") : secret "dns-default-metrics-tls" not found
Jul 23 21:52:25.289015 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:25.288991    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 21:52:25.289015 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:25.289011    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm podName:2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:29.288993613 +0000 UTC m=+44.679386218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jh4m" (UID: "2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5") : configmap "kube-rbac-proxy" not found
Jul 23 21:52:25.289198 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:25.289062    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls podName:2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:29.289046179 +0000 UTC m=+44.679438789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jh4m" (UID: "2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5") : secret "insights-runtime-extractor-tls" not found
Jul 23 21:52:25.433421 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:25.433378    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:25.526799 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:25.526760    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:25.625415 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:25.625329    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:26.241657 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:26.241627    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:26.639745 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:26.639657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:27.041982 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:27.041953    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:27.440425 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:27.440338    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:27.836833 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:27.836804    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:28.038445 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:28.038409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:28.432447 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:28.432411    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:28.644526 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:28.644488    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:28.742899 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:28.742838    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:29.092440 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:29.092343    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:29.317883 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:29.317828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:29.317883 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:29.317885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:29.318130 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:29.317912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:29.318130 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:29.317983    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 21:52:29.318130 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:29.317999    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 21:52:29.318130 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:29.318036    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm podName:2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:37.318023599 +0000 UTC m=+52.708416193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5jh4m" (UID: "2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5") : configmap "kube-rbac-proxy" not found
Jul 23 21:52:29.318130 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:29.318062    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls podName:2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:37.318045293 +0000 UTC m=+52.708437905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5jh4m" (UID: "2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5") : secret "insights-runtime-extractor-tls" not found
Jul 23 21:52:29.321921 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:29.321891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/59368801-3baf-4007-bf31-62e00a4a8306-metrics-tls\") pod \"dns-default-l5bhg\" (UID: \"59368801-3baf-4007-bf31-62e00a4a8306\") " pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:29.332436 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:29.332410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:29.435941 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:29.435838    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:29.490264 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:29.490229    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l5bhg"]
Jul 23 21:52:29.494225 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:52:29.494198    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59368801_3baf_4007_bf31_62e00a4a8306.slice/crio-1052d2adff8404e7cfcd7def3ac32c307ebc444ae340388b8fe4579e0cf2aa86 WatchSource:0}: Error finding container 1052d2adff8404e7cfcd7def3ac32c307ebc444ae340388b8fe4579e0cf2aa86: Status 404 returned error can't find the container with id 1052d2adff8404e7cfcd7def3ac32c307ebc444ae340388b8fe4579e0cf2aa86
Jul 23 21:52:29.498113 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:29.498088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:29.846674 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:29.846638    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:30.034791 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:30.034750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:30.335984 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:30.335952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l5bhg" event={"ID":"59368801-3baf-4007-bf31-62e00a4a8306","Type":"ContainerStarted","Data":"1052d2adff8404e7cfcd7def3ac32c307ebc444ae340388b8fe4579e0cf2aa86"}
Jul 23 21:52:30.638361 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:30.638245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:31.055147 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:31.055103    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:31.258555 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:31.258519    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:31.440181 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:31.440139    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:31.841014 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:31.840979    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:32.047955 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:32.047913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:32.342007 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:32.341930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l5bhg" event={"ID":"59368801-3baf-4007-bf31-62e00a4a8306","Type":"ContainerStarted","Data":"f8744d97d662de23bc4d63ead0171a5b0f74b604afcb49de29e3af7177190fb2"}
Jul 23 21:52:32.342007 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:32.341978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l5bhg" event={"ID":"59368801-3baf-4007-bf31-62e00a4a8306","Type":"ContainerStarted","Data":"417f89566c3c661311c43d04622dc8b9efb747d93ddcef18e951cd1d7c6e3352"}
Jul 23 21:52:32.342189 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:32.342056    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:32.359915 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:32.359619    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l5bhg" podStartSLOduration=9.508990793 podStartE2EDuration="11.359604161s" totalImagesPullingTime="1.850613368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:32.35867698 +0000 UTC m=+47.749069610" watchObservedRunningTime="2026-07-23 21:52:32.359604161 +0000 UTC m=+47.749996776"
Jul 23 21:52:32.638556 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:32.638462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:33.050330 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:33.050302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:33.284569 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:33.284539    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:33.786103 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:33.786064    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:33.881745 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:33.881713    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:34.117184 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:34.117067    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:34.262938 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:34.262903    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:34.643800 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:34.643753    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:34.834424 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:34.834388    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:35.250878 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:35.250814    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:35.441274 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:35.441242    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:37.383582 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:37.383538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:37.383582 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:37.383583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:37.384157 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:37.384138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:37.386068 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:37.386047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5jh4m\" (UID: \"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5\") " pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:37.468835 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:37.468803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5jh4m"
Jul 23 21:52:37.626169 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:37.626143    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5jh4m"]
Jul 23 21:52:37.628432 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:52:37.628399    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db9d3b8_fd1b_42d9_96ee_9eae80cfdbe5.slice/crio-1ff2cd5884fd91d017724b80510e89eec46df5620e860de8d6eaee1e0dcbc299 WatchSource:0}: Error finding container 1ff2cd5884fd91d017724b80510e89eec46df5620e860de8d6eaee1e0dcbc299: Status 404 returned error can't find the container with id 1ff2cd5884fd91d017724b80510e89eec46df5620e860de8d6eaee1e0dcbc299
Jul 23 21:52:38.357337 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:38.357294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jh4m" event={"ID":"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5","Type":"ContainerStarted","Data":"1ff2cd5884fd91d017724b80510e89eec46df5620e860de8d6eaee1e0dcbc299"}
Jul 23 21:52:39.089204 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:39.089171    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:39.189098 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:39.189061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:39.292333 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:39.292286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:39.391128 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:39.391039    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:39.569429 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:39.569367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:39.759276 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:39.759229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:39.862060 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:39.862017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:40.163452 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.163360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:40.233687 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.233659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:40.786289 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.786246    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"]
Jul 23 21:52:40.893786 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.893746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:40.898695 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.898231    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"]
Jul 23 21:52:40.898695 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.898328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:40.902901 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.902639    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 23 21:52:40.902901 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.902725    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 23 21:52:40.903128 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.903105    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 23 21:52:40.903441 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.903357    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 23 21:52:40.935375 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.935334    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"]
Jul 23 21:52:40.935375 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.935380    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"]
Jul 23 21:52:40.935604 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.935502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:40.937911 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.937828    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 23 21:52:40.938075 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.937975    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 23 21:52:40.938075 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.938037    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 23 21:52:40.938630 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.938604    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 23 21:52:40.990950 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:40.990906    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:41.012820 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.012784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bbcn\" (UniqueName: \"kubernetes.io/projected/24c25bbb-bcee-4bf7-b316-e22a28e827ce-kube-api-access-9bbcn\") pod \"klusterlet-addon-workmgr-64fbdd4859-flr4r\" (UID: \"24c25bbb-bcee-4bf7-b316-e22a28e827ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:41.013016 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.012876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c25bbb-bcee-4bf7-b316-e22a28e827ce-tmp\") pod \"klusterlet-addon-workmgr-64fbdd4859-flr4r\" (UID: \"24c25bbb-bcee-4bf7-b316-e22a28e827ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:41.013016 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.012949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/24c25bbb-bcee-4bf7-b316-e22a28e827ce-klusterlet-config\") pod \"klusterlet-addon-workmgr-64fbdd4859-flr4r\" (UID: \"24c25bbb-bcee-4bf7-b316-e22a28e827ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:41.082479 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.082378    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:41.113771 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.113730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-ca\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.113990 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.113892    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bbcn\" (UniqueName: \"kubernetes.io/projected/24c25bbb-bcee-4bf7-b316-e22a28e827ce-kube-api-access-9bbcn\") pod \"klusterlet-addon-workmgr-64fbdd4859-flr4r\" (UID: \"24c25bbb-bcee-4bf7-b316-e22a28e827ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:41.113990 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.113928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4415ab83-3890-4935-be81-b71337a396a0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.113990 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.113980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-hub\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.114241 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.114115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.114241 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.114137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.114241 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.114174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c25bbb-bcee-4bf7-b316-e22a28e827ce-tmp\") pod \"klusterlet-addon-workmgr-64fbdd4859-flr4r\" (UID: \"24c25bbb-bcee-4bf7-b316-e22a28e827ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:41.114241 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.114212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfcqr\" (UniqueName: \"kubernetes.io/projected/4415ab83-3890-4935-be81-b71337a396a0-kube-api-access-wfcqr\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.114440 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.114288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/24c25bbb-bcee-4bf7-b316-e22a28e827ce-klusterlet-config\") pod \"klusterlet-addon-workmgr-64fbdd4859-flr4r\" (UID: \"24c25bbb-bcee-4bf7-b316-e22a28e827ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:41.114561 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.114536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24c25bbb-bcee-4bf7-b316-e22a28e827ce-tmp\") pod \"klusterlet-addon-workmgr-64fbdd4859-flr4r\" (UID: \"24c25bbb-bcee-4bf7-b316-e22a28e827ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:41.117319 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.117294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/24c25bbb-bcee-4bf7-b316-e22a28e827ce-klusterlet-config\") pod \"klusterlet-addon-workmgr-64fbdd4859-flr4r\" (UID: \"24c25bbb-bcee-4bf7-b316-e22a28e827ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:41.125098 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.125065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bbcn\" (UniqueName: \"kubernetes.io/projected/24c25bbb-bcee-4bf7-b316-e22a28e827ce-kube-api-access-9bbcn\") pod \"klusterlet-addon-workmgr-64fbdd4859-flr4r\" (UID: \"24c25bbb-bcee-4bf7-b316-e22a28e827ce\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:41.186775 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.186736    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:41.214463 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.214415    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:41.214730 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.214707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4415ab83-3890-4935-be81-b71337a396a0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.214803 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.214762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-hub\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.214803 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.214791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.214938 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.214814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.216062 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.215345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfcqr\" (UniqueName: \"kubernetes.io/projected/4415ab83-3890-4935-be81-b71337a396a0-kube-api-access-wfcqr\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.216062 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.215422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-ca\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.216062 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.215466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4415ab83-3890-4935-be81-b71337a396a0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.217974 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.217917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-hub\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.218105 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.217985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.218105 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.218089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.218388 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.218365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4415ab83-3890-4935-be81-b71337a396a0-ca\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.226977 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.226938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfcqr\" (UniqueName: \"kubernetes.io/projected/4415ab83-3890-4935-be81-b71337a396a0-kube-api-access-wfcqr\") pod \"cluster-proxy-proxy-agent-5bddd5d987-4vsd6\" (UID: \"4415ab83-3890-4935-be81-b71337a396a0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.261111 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.261075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"
Jul 23 21:52:41.284160 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.284126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:41.794697 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.794661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:41.856900 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:41.856842    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:42.206540 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:42.206505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:42.307905 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:42.307852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:42.347106 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:42.347079    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l5bhg"
Jul 23 21:52:43.346363 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:43.346324    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6"]
Jul 23 21:52:43.349963 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:52:43.349932    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4415ab83_3890_4935_be81_b71337a396a0.slice/crio-0e57b3c07374219af88b36ad650255571404e36f44681c345ce20eedb98eda97 WatchSource:0}: Error finding container 0e57b3c07374219af88b36ad650255571404e36f44681c345ce20eedb98eda97: Status 404 returned error can't find the container with id 0e57b3c07374219af88b36ad650255571404e36f44681c345ce20eedb98eda97
Jul 23 21:52:43.361687 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:43.361656    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"]
Jul 23 21:52:43.364911 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:52:43.364873    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c25bbb_bcee_4bf7_b316_e22a28e827ce.slice/crio-08bcb51d0dd4e93eb4612fc018133e5bab24f97df0dffe4358105f0e1a995aaf WatchSource:0}: Error finding container 08bcb51d0dd4e93eb4612fc018133e5bab24f97df0dffe4358105f0e1a995aaf: Status 404 returned error can't find the container with id 08bcb51d0dd4e93eb4612fc018133e5bab24f97df0dffe4358105f0e1a995aaf
Jul 23 21:52:43.371794 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:43.371766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r" event={"ID":"24c25bbb-bcee-4bf7-b316-e22a28e827ce","Type":"ContainerStarted","Data":"08bcb51d0dd4e93eb4612fc018133e5bab24f97df0dffe4358105f0e1a995aaf"}
Jul 23 21:52:43.373071 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:43.373041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6" event={"ID":"4415ab83-3890-4935-be81-b71337a396a0","Type":"ContainerStarted","Data":"0e57b3c07374219af88b36ad650255571404e36f44681c345ce20eedb98eda97"}
Jul 23 21:52:44.389622 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:44.389576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jh4m" event={"ID":"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5","Type":"ContainerStarted","Data":"778228c27c24119692f0d25e764031883ebb6921b95396b517b721e364fcb6ac"}
Jul 23 21:52:44.495549 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:44.495487    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:44.560851 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:44.560798    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:44.697896 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:44.697773    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:44.792138 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:44.792006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:45.358699 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:45.358628    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:46.302761 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:46.302391    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rs2g2"
Jul 23 21:52:46.421294 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:46.421229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:46.522631 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:46.522574    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:46.682484 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:46.682357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:46.802309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:46.802264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:46.901451 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:46.901413    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:48.709230 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:48.709197    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:48.773586 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:48.773480    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:48.941786 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:48.941746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:50.002227 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.002179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:50.623213 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.623168    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:50.793562 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.793530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:50.795851 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.795685    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 21:52:50.806728 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.806699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e38142be-62e8-49b7-af8d-cad1fcb9b7c5-original-pull-secret\") pod \"global-pull-secret-syncer-455s5\" (UID: \"e38142be-62e8-49b7-af8d-cad1fcb9b7c5\") " pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:50.894630 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.894549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:50.894630 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.894604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdqc\" (UniqueName: \"kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc\") pod \"network-check-target-zk875\" (UID: \"dfa41a7c-b457-473c-a1cf-5d9eac128eea\") " pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:50.897322 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.897289    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 21:52:50.901037 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.898161    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 21:52:50.907615 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.907587    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 21:52:50.907615 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.907607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e81cf8-335f-4794-a193-449e38604f3c-metrics-certs\") pod \"network-metrics-daemon-szssk\" (UID: \"f7e81cf8-335f-4794-a193-449e38604f3c\") " pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:50.917997 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.917965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtdqc\" (UniqueName: \"kubernetes.io/projected/dfa41a7c-b457-473c-a1cf-5d9eac128eea-kube-api-access-qtdqc\") pod \"network-check-target-zk875\" (UID: \"dfa41a7c-b457-473c-a1cf-5d9eac128eea\") " pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:50.918166 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.918148    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fxtft\""
Jul 23 21:52:50.923174 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.923155    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-n9zwh\""
Jul 23 21:52:50.925777 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.925759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-szssk"
Jul 23 21:52:50.931929 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:50.931906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:51.021316 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.021275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-455s5"
Jul 23 21:52:51.075576 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.075495    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-szssk"]
Jul 23 21:52:51.078365 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:52:51.078325    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e81cf8_335f_4794_a193_449e38604f3c.slice/crio-154c1e88b814865e258151bebdc7de1f256e715256f285abe0399c8d7082bcca WatchSource:0}: Error finding container 154c1e88b814865e258151bebdc7de1f256e715256f285abe0399c8d7082bcca: Status 404 returned error can't find the container with id 154c1e88b814865e258151bebdc7de1f256e715256f285abe0399c8d7082bcca
Jul 23 21:52:51.119227 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.119185    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zk875"]
Jul 23 21:52:51.123247 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:52:51.123217    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa41a7c_b457_473c_a1cf_5d9eac128eea.slice/crio-6b68cc1b429a8a0595f8f6362714087e505588a671c4996d10696818c7bc2f48 WatchSource:0}: Error finding container 6b68cc1b429a8a0595f8f6362714087e505588a671c4996d10696818c7bc2f48: Status 404 returned error can't find the container with id 6b68cc1b429a8a0595f8f6362714087e505588a671c4996d10696818c7bc2f48
Jul 23 21:52:51.194426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.194335    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-455s5"]
Jul 23 21:52:51.198288 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:52:51.198258    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38142be_62e8_49b7_af8d_cad1fcb9b7c5.slice/crio-4561d7d5d11cbc841c4e53ebc59b80ccea1398a9123f286d770dea7977756c1e WatchSource:0}: Error finding container 4561d7d5d11cbc841c4e53ebc59b80ccea1398a9123f286d770dea7977756c1e: Status 404 returned error can't find the container with id 4561d7d5d11cbc841c4e53ebc59b80ccea1398a9123f286d770dea7977756c1e
Jul 23 21:52:51.380493 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.380453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:51.413452 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.413418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zk875" event={"ID":"dfa41a7c-b457-473c-a1cf-5d9eac128eea","Type":"ContainerStarted","Data":"6b68cc1b429a8a0595f8f6362714087e505588a671c4996d10696818c7bc2f48"}
Jul 23 21:52:51.414761 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.414732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-szssk" event={"ID":"f7e81cf8-335f-4794-a193-449e38604f3c","Type":"ContainerStarted","Data":"154c1e88b814865e258151bebdc7de1f256e715256f285abe0399c8d7082bcca"}
Jul 23 21:52:51.416309 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.416280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-455s5" event={"ID":"e38142be-62e8-49b7-af8d-cad1fcb9b7c5","Type":"ContainerStarted","Data":"4561d7d5d11cbc841c4e53ebc59b80ccea1398a9123f286d770dea7977756c1e"}
Jul 23 21:52:51.418969 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.418940    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6" event={"ID":"4415ab83-3890-4935-be81-b71337a396a0","Type":"ContainerStarted","Data":"05d6b28c2be2ae0bec2cb8c757127aeda1ee35f856196b9fa8091412722c41f4"}
Jul 23 21:52:51.419105 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.418987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6" event={"ID":"4415ab83-3890-4935-be81-b71337a396a0","Type":"ContainerStarted","Data":"576382c6c72ac4783d3a767e5ab2661ace1026089c18fa376fffcc08e709a34d"}
Jul 23 21:52:51.419105 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.419006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6" event={"ID":"4415ab83-3890-4935-be81-b71337a396a0","Type":"ContainerStarted","Data":"8844338164a5905f94aa81483167636811c6e32b85141f149b24b1d0c88c07be"}
Jul 23 21:52:51.421243 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.421221    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jh4m" event={"ID":"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5","Type":"ContainerStarted","Data":"f08bf91142bae9d326ccb49b545978739939bec03914867b2c20e2c9e4d7aaaf"}
Jul 23 21:52:51.422538 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.422515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r" event={"ID":"24c25bbb-bcee-4bf7-b316-e22a28e827ce","Type":"ContainerStarted","Data":"ff89131b305587666b56037ef2f788c3a484dce1827ba37ae002f363171397c6"}
Jul 23 21:52:51.422761 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.422739    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:51.424592 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.424572    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r"
Jul 23 21:52:51.444825 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.444718    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5bddd5d987-4vsd6" podStartSLOduration=4.137648741 podStartE2EDuration="11.444694666s" totalImagesPullingTime="7.307045925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:51.443624348 +0000 UTC m=+66.834016965" watchObservedRunningTime="2026-07-23 21:52:51.444694666 +0000 UTC m=+66.835087283"
Jul 23 21:52:51.462809 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:51.462734    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64fbdd4859-flr4r" podStartSLOduration=4.151310497 podStartE2EDuration="11.462713227s" totalImagesPullingTime="7.31140273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:51.462442688 +0000 UTC m=+66.852835305" watchObservedRunningTime="2026-07-23 21:52:51.462713227 +0000 UTC m=+66.853105838"
Jul 23 21:52:52.164207 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.164048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:52.237979 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.237931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:52.409755 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.409324    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9d597dcfc-n9whk"]
Jul 23 21:52:52.413923 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.413053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.416445 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.415947    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 23 21:52:52.416923 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.416794    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 23 21:52:52.417690 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.417251    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 23 21:52:52.417690 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.417411    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 23 21:52:52.417690 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.417445    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 23 21:52:52.417690 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.417452    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 23 21:52:52.417690 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.417656    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 23 21:52:52.418115 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.417990    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-srfr5\""
Jul 23 21:52:52.418258 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.418233    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:52.423420 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.423393    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 23 21:52:52.430248 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.429641    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d597dcfc-n9whk"]
Jul 23 21:52:52.489526 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.489428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:52.509471 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.508018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-trusted-ca-bundle\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.509471 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.508102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-oauth-config\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.509471 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.508217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-oauth-serving-cert\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.509471 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.508321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl2c7\" (UniqueName: \"kubernetes.io/projected/c8a580c9-abfb-4255-8094-60b5e7bc109c-kube-api-access-bl2c7\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.509471 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.508392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-config\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.509471 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.508423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-service-ca\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.509471 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.508463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-serving-cert\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.609470 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.609374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-trusted-ca-bundle\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.609470 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.609431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-oauth-config\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.609710 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.609484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-oauth-serving-cert\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.609710 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.609507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl2c7\" (UniqueName: \"kubernetes.io/projected/c8a580c9-abfb-4255-8094-60b5e7bc109c-kube-api-access-bl2c7\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.609710 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.609542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-config\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.609710 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.609572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-service-ca\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.609710 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.609598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-serving-cert\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.610776 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.610747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-oauth-serving-cert\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.613380 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.611656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-trusted-ca-bundle\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.614606 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.613826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-serving-cert\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.614606 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.614571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-service-ca\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.614762 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.614733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-config\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.616538 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.616460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-oauth-config\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.625705 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.625616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl2c7\" (UniqueName: \"kubernetes.io/projected/c8a580c9-abfb-4255-8094-60b5e7bc109c-kube-api-access-bl2c7\") pod \"console-9d597dcfc-n9whk\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") " pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.741609 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.741569    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:52:52.951809 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.951772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:52.972295 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:52.970600    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d597dcfc-n9whk"]
Jul 23 21:52:52.975106 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:52:52.974414    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a580c9_abfb_4255_8094_60b5e7bc109c.slice/crio-ead1ac909de700dfefd7c762f9b7da4f6942e6e589ad643a571672ff62e1bdd3 WatchSource:0}: Error finding container ead1ac909de700dfefd7c762f9b7da4f6942e6e589ad643a571672ff62e1bdd3: Status 404 returned error can't find the container with id ead1ac909de700dfefd7c762f9b7da4f6942e6e589ad643a571672ff62e1bdd3
Jul 23 21:52:53.181594 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.181454    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:53.288500 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.287088    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v7wch"]
Jul 23 21:52:53.292652 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.292142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.299032 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.298946    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 21:52:53.302920 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.302892    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ktmzs\""
Jul 23 21:52:53.303077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.302975    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 21:52:53.303212 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.303184    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 21:52:53.303212 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.303197    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 21:52:53.303426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.303403    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 21:52:53.303563 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.303500    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 21:52:53.315742 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.315712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a29f1bfa-eccf-4f32-8209-44086ce2bf58-root\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.316007 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.315760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-wtmp\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.316007 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.315790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-tls\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.316007 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.315847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a29f1bfa-eccf-4f32-8209-44086ce2bf58-sys\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.316007 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.315893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.316007 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.315930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfrb5\" (UniqueName: \"kubernetes.io/projected/a29f1bfa-eccf-4f32-8209-44086ce2bf58-kube-api-access-tfrb5\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.316007 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.315959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a29f1bfa-eccf-4f32-8209-44086ce2bf58-metrics-client-ca\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.316270 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.316022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-accelerators-collector-config\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.316270 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.316112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-textfile\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.321426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.321290    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.416997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a29f1bfa-eccf-4f32-8209-44086ce2bf58-root\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-wtmp\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-tls\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a29f1bfa-eccf-4f32-8209-44086ce2bf58-sys\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfrb5\" (UniqueName: \"kubernetes.io/projected/a29f1bfa-eccf-4f32-8209-44086ce2bf58-kube-api-access-tfrb5\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a29f1bfa-eccf-4f32-8209-44086ce2bf58-metrics-client-ca\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-accelerators-collector-config\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-textfile\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a29f1bfa-eccf-4f32-8209-44086ce2bf58-sys\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a29f1bfa-eccf-4f32-8209-44086ce2bf58-root\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-wtmp\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:53.417676    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.417682    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-textfile\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.417895 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:52:53.417732    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-tls podName:a29f1bfa-eccf-4f32-8209-44086ce2bf58 nodeName:}" failed. No retries permitted until 2026-07-23 21:52:53.917713697 +0000 UTC m=+69.308106291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-tls") pod "node-exporter-v7wch" (UID: "a29f1bfa-eccf-4f32-8209-44086ce2bf58") : secret "node-exporter-tls" not found
Jul 23 21:52:53.418754 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.418223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a29f1bfa-eccf-4f32-8209-44086ce2bf58-metrics-client-ca\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.418754 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.418384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-accelerators-collector-config\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.421054 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.420956    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:53.421709 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.421619    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.426802 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.426747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfrb5\" (UniqueName: \"kubernetes.io/projected/a29f1bfa-eccf-4f32-8209-44086ce2bf58-kube-api-access-tfrb5\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.444458 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.444371    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-szssk" event={"ID":"f7e81cf8-335f-4794-a193-449e38604f3c","Type":"ContainerStarted","Data":"6483b7df8f673f5299ef6f75dbba9b2af6236c38802f0bd0376b0938ac2772d5"}
Jul 23 21:52:53.444458 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.444444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-szssk" event={"ID":"f7e81cf8-335f-4794-a193-449e38604f3c","Type":"ContainerStarted","Data":"689c0834bf293df540a714cf6aac090f3fa9daa405a35fb785cdf01b1872cb63"}
Jul 23 21:52:53.447318 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.447279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d597dcfc-n9whk" event={"ID":"c8a580c9-abfb-4255-8094-60b5e7bc109c","Type":"ContainerStarted","Data":"ead1ac909de700dfefd7c762f9b7da4f6942e6e589ad643a571672ff62e1bdd3"}
Jul 23 21:52:53.464660 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.464604    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-szssk" podStartSLOduration=67.213723031 podStartE2EDuration="1m8.464586058s" totalImagesPullingTime="1.250863027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:53.463778525 +0000 UTC m=+68.854171143" watchObservedRunningTime="2026-07-23 21:52:53.464586058 +0000 UTC m=+68.854978673"
Jul 23 21:52:53.520340 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.520305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:53.618528 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.618489    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:53.718110 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.718030    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:53.819890 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.819720    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:53.918416 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.918323    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:53.920356 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.920328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-tls\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:53.923542 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:53.923513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a29f1bfa-eccf-4f32-8209-44086ce2bf58-node-exporter-tls\") pod \"node-exporter-v7wch\" (UID: \"a29f1bfa-eccf-4f32-8209-44086ce2bf58\") " pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:54.152302 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:54.152226    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:54.205267 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:54.205230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v7wch"
Jul 23 21:52:54.286443 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:54.286409    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:54.807572 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:54.807522    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:56.210163 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:56.210129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:56.309247 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:56.309168    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:56.573423 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:56.573329    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:56.669482 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:56.669445    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:57.346704 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:57.346639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:57.449254 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:57.449164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:57.962592 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:57.962550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:58.047463 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:58.047428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:58.070424 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:52:58.070307    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29f1bfa_eccf_4f32_8209_44086ce2bf58.slice/crio-e1ad1c7fefc527b17f078cc7dbea21989db95dd2da6c1596248aa335874ff4c9 WatchSource:0}: Error finding container e1ad1c7fefc527b17f078cc7dbea21989db95dd2da6c1596248aa335874ff4c9: Status 404 returned error can't find the container with id e1ad1c7fefc527b17f078cc7dbea21989db95dd2da6c1596248aa335874ff4c9
Jul 23 21:52:58.246316 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:58.246216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:58.468813 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:58.468779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-455s5" event={"ID":"e38142be-62e8-49b7-af8d-cad1fcb9b7c5","Type":"ContainerStarted","Data":"93b7fedae534efabb05f251f66587311129ffb2a247bd064bf0afc6ec266282d"}
Jul 23 21:52:58.470101 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:58.470070    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v7wch" event={"ID":"a29f1bfa-eccf-4f32-8209-44086ce2bf58","Type":"ContainerStarted","Data":"e1ad1c7fefc527b17f078cc7dbea21989db95dd2da6c1596248aa335874ff4c9"}
Jul 23 21:52:58.472660 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:58.472631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5jh4m" event={"ID":"2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5","Type":"ContainerStarted","Data":"c6b6b4c30c07e72f4c6670e962a81241a8b2b33619ba91c9bfc63af701ed9bf9"}
Jul 23 21:52:58.474716 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:58.474680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zk875" event={"ID":"dfa41a7c-b457-473c-a1cf-5d9eac128eea","Type":"ContainerStarted","Data":"2e0dac1807b996c901b87794906a32b358ba9e8cea5105837a100716d1c33179"}
Jul 23 21:52:58.475007 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:58.474990    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:52:58.504686 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:58.503689    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-455s5" podStartSLOduration=66.622653599 podStartE2EDuration="1m13.503669867s" totalImagesPullingTime="6.881016268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:58.48688629 +0000 UTC m=+73.877278904" watchObservedRunningTime="2026-07-23 21:52:58.503669867 +0000 UTC m=+73.894062504"
Jul 23 21:52:58.504686 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:58.504511    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zk875" podStartSLOduration=66.54699927 podStartE2EDuration="1m13.504492139s" totalImagesPullingTime="6.957492869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:58.502850127 +0000 UTC m=+73.893242743" watchObservedRunningTime="2026-07-23 21:52:58.504492139 +0000 UTC m=+73.894884759"
Jul 23 21:52:58.520658 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:58.520566    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5jh4m" podStartSLOduration=17.261518341 podStartE2EDuration="37.520546921s" totalImagesPullingTime="20.25902858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:52:58.52020322 +0000 UTC m=+73.910595836" watchObservedRunningTime="2026-07-23 21:52:58.520546921 +0000 UTC m=+73.910939537"
Jul 23 21:52:59.513497 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.512513    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 21:52:59.515543 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.515508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.519770 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.519745    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 21:52:59.519988 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.519966    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 21:52:59.520080 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.519968    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-akrnbk3qbtfun\""
Jul 23 21:52:59.522653 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.522506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 21:52:59.522653 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.522541    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 21:52:59.522653 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.522579    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 21:52:59.523511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.523202    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 21:52:59.523511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.523385    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 21:52:59.523511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.523406    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 21:52:59.526187 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.524831    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 21:52:59.526187 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.525105    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 21:52:59.526187 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.525225    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cxkvl\""
Jul 23 21:52:59.526187 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.525403    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 21:52:59.526187 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.525570    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 21:52:59.526187 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.525696    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 21:52:59.526187 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.525744    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 21:52:59.536634 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.536609    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 21:52:59.552729 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.552690    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 21:52:59.563266 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-config\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563424 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9mn9\" (UniqueName: \"kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-kube-api-access-d9mn9\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563424 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563424 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563424 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563424 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563424 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563734 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563734 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563734 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563734 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563734 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563734 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563734 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-config-out\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.563734 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.564133 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.564133 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-web-config\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.564133 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.564133 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.563899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.564385 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.564354    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.572296 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.572261    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:52:59.665645 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.665608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.665843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.665659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.665843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.665805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.665999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.665847    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.665999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.665896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.665999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.665920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.665999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.665964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.665999 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.665989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666223 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666223 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666223 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666223 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-config-out\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666223 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666223 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666223 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-web-config\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666223 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666615 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666615 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666615 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-config\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.666615 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9mn9\" (UniqueName: \"kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-kube-api-access-d9mn9\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.667008 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.666689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.667463 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.667411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.667679 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.667575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.668941 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.667893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.668941 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.668436    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.669113 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.669055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.670207 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.670183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.671556 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.671443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.672323 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.672273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.673251 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.673230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.674235 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.674190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.674310 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.674250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.675415 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.675331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.676177 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.676153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-web-config\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.676434 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.676413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.676651 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.676629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-config-out\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.676802 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.676780    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.677121 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.677103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.678703 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.678679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-config\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.691026 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.690919    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9mn9\" (UniqueName: \"kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-kube-api-access-d9mn9\") pod \"prometheus-k8s-0\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.827298 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.827255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:52:59.857735 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:52:59.857703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:00.047489 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:00.047453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 21:53:00.051523 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:53:00.051492    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea513b0f_adfe_4b97_9c10_917880a20b99.slice/crio-c42167bdd5b8d4aaa635d783756854c37950333aa16af8d9e10e6ebda734c2e5 WatchSource:0}: Error finding container c42167bdd5b8d4aaa635d783756854c37950333aa16af8d9e10e6ebda734c2e5: Status 404 returned error can't find the container with id c42167bdd5b8d4aaa635d783756854c37950333aa16af8d9e10e6ebda734c2e5
Jul 23 21:53:00.491509 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:00.491468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d597dcfc-n9whk" event={"ID":"c8a580c9-abfb-4255-8094-60b5e7bc109c","Type":"ContainerStarted","Data":"9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a"}
Jul 23 21:53:00.492960 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:00.492929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerStarted","Data":"c42167bdd5b8d4aaa635d783756854c37950333aa16af8d9e10e6ebda734c2e5"}
Jul 23 21:53:00.494791 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:00.494762    2579 generic.go:356] "Generic (PLEG): container finished" podID="a29f1bfa-eccf-4f32-8209-44086ce2bf58" containerID="c6aabb75b5af143765ab49ad883c2de75eb3c174f2ddaf3f603e8b98f555517a" exitCode=0
Jul 23 21:53:00.494910 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:00.494834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v7wch" event={"ID":"a29f1bfa-eccf-4f32-8209-44086ce2bf58","Type":"ContainerDied","Data":"c6aabb75b5af143765ab49ad883c2de75eb3c174f2ddaf3f603e8b98f555517a"}
Jul 23 21:53:00.511233 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:00.511174    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9d597dcfc-n9whk" podStartSLOduration=2.053397148 podStartE2EDuration="8.511152671s" totalImagesPullingTime="6.457755523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:00.50949766 +0000 UTC m=+75.899890275" watchObservedRunningTime="2026-07-23 21:53:00.511152671 +0000 UTC m=+75.901545287"
Jul 23 21:53:01.251847 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:01.251809    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:01.342954 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:01.342920    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:01.502582 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:01.502495    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerID="5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f" exitCode=0
Jul 23 21:53:01.502746 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:01.502601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerDied","Data":"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"}
Jul 23 21:53:01.504916 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:01.504892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v7wch" event={"ID":"a29f1bfa-eccf-4f32-8209-44086ce2bf58","Type":"ContainerStarted","Data":"3bd34306e895722fcea05f366ff274845c9fee82dc49977a645fdc266964c171"}
Jul 23 21:53:01.505036 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:01.504947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v7wch" event={"ID":"a29f1bfa-eccf-4f32-8209-44086ce2bf58","Type":"ContainerStarted","Data":"24fed01cb8b802fd0fba30fd4cd1ddf02dd4ed82fd8b775eb6bbef02cc27ecb5"}
Jul 23 21:53:01.556034 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:01.555969    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v7wch" podStartSLOduration=6.623098293 podStartE2EDuration="8.555950969s" totalImagesPullingTime="1.853128509s" totalInitContainerRuntime="79.724167ms" isStatefulPod=true podCreationTimestamp="2026-07-23 21:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:01.554782707 +0000 UTC m=+76.945175323" watchObservedRunningTime="2026-07-23 21:53:01.555950969 +0000 UTC m=+76.946343586"
Jul 23 21:53:02.727369 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:02.727329    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:02.743075 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:02.743035    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:53:02.743291 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:02.743104    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:53:02.749190 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:02.749161    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:53:02.805242 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:02.804767    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:03.518189 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:03.518155    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:53:05.524410 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:05.524266    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerStarted","Data":"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"}
Jul 23 21:53:05.524410 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:05.524395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerStarted","Data":"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"}
Jul 23 21:53:06.531113 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:06.531073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerStarted","Data":"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"}
Jul 23 21:53:06.531559 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:06.531182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerStarted","Data":"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"}
Jul 23 21:53:06.531559 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:06.531227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerStarted","Data":"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"}
Jul 23 21:53:06.531559 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:06.531278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerStarted","Data":"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"}
Jul 23 21:53:06.568218 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:06.568158    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.195434364 podStartE2EDuration="7.568143119s" totalImagesPullingTime="5.3527844s" totalInitContainerRuntime="19.924355ms" isStatefulPod=false podCreationTimestamp="2026-07-23 21:52:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:53:06.567114952 +0000 UTC m=+81.957507569" watchObservedRunningTime="2026-07-23 21:53:06.568143119 +0000 UTC m=+81.958535734"
Jul 23 21:53:09.315815 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:09.315777    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d597dcfc-n9whk"]
Jul 23 21:53:09.827982 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:09.827940    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:53:10.003087 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:10.003056    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:10.103771 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:10.103682    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:12.188875 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:12.188823    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:12.261130 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:12.261093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:13.055473 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:13.055441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:13.129038 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:13.129006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:14.126715 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:14.126679    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:14.221964 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:14.221925    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:15.806012 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:15.805955    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:20.010650 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:20.010602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:20.109472 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:20.109437    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:20.181213 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:20.181179    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:27.168276 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:27.168243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:27.224771 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:27.224734    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:29.481937 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:29.481905    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zk875"
Jul 23 21:53:34.341949 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.341856    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9d597dcfc-n9whk" podUID="c8a580c9-abfb-4255-8094-60b5e7bc109c" containerName="console" containerID="cri-o://9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a" gracePeriod=15
Jul 23 21:53:34.585838 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.585811    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:53:34.632959 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.632842    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8a580c9-abfb-4255-8094-60b5e7bc109c" containerID="9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a" exitCode=0
Jul 23 21:53:34.632959 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.632920    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d597dcfc-n9whk"
Jul 23 21:53:34.632959 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.632932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d597dcfc-n9whk" event={"ID":"c8a580c9-abfb-4255-8094-60b5e7bc109c","Type":"ContainerDied","Data":"9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a"}
Jul 23 21:53:34.633162 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.632992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d597dcfc-n9whk" event={"ID":"c8a580c9-abfb-4255-8094-60b5e7bc109c","Type":"ContainerDied","Data":"ead1ac909de700dfefd7c762f9b7da4f6942e6e589ad643a571672ff62e1bdd3"}
Jul 23 21:53:34.633162 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.633016    2579 scope.go:117] "RemoveContainer" containerID="9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a"
Jul 23 21:53:34.641683 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.641662    2579 scope.go:117] "RemoveContainer" containerID="9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a"
Jul 23 21:53:34.641997 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:53:34.641973    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a\": container with ID starting with 9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a not found: ID does not exist" containerID="9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a"
Jul 23 21:53:34.642055 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.642010    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a"} err="failed to get container status \"9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a\": rpc error: code = NotFound desc = could not find container \"9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a\": container with ID starting with 9ab0743d77ae43c2a779976b1816bc25745e368c532b9d504e406d1ffe9dda3a not found: ID does not exist"
Jul 23 21:53:34.669383 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.669352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-oauth-serving-cert\") pod \"c8a580c9-abfb-4255-8094-60b5e7bc109c\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") "
Jul 23 21:53:34.669383 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.669386    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-oauth-config\") pod \"c8a580c9-abfb-4255-8094-60b5e7bc109c\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") "
Jul 23 21:53:34.669624 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.669420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-config\") pod \"c8a580c9-abfb-4255-8094-60b5e7bc109c\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") "
Jul 23 21:53:34.669624 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.669439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-service-ca\") pod \"c8a580c9-abfb-4255-8094-60b5e7bc109c\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") "
Jul 23 21:53:34.669624 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.669467    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-serving-cert\") pod \"c8a580c9-abfb-4255-8094-60b5e7bc109c\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") "
Jul 23 21:53:34.669624 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.669520    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl2c7\" (UniqueName: \"kubernetes.io/projected/c8a580c9-abfb-4255-8094-60b5e7bc109c-kube-api-access-bl2c7\") pod \"c8a580c9-abfb-4255-8094-60b5e7bc109c\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") "
Jul 23 21:53:34.669624 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.669540    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-trusted-ca-bundle\") pod \"c8a580c9-abfb-4255-8094-60b5e7bc109c\" (UID: \"c8a580c9-abfb-4255-8094-60b5e7bc109c\") "
Jul 23 21:53:34.669949 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.669909    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c8a580c9-abfb-4255-8094-60b5e7bc109c" (UID: "c8a580c9-abfb-4255-8094-60b5e7bc109c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 21:53:34.670077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.669914    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-config" (OuterVolumeSpecName: "console-config") pod "c8a580c9-abfb-4255-8094-60b5e7bc109c" (UID: "c8a580c9-abfb-4255-8094-60b5e7bc109c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 21:53:34.670077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.669962    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-service-ca" (OuterVolumeSpecName: "service-ca") pod "c8a580c9-abfb-4255-8094-60b5e7bc109c" (UID: "c8a580c9-abfb-4255-8094-60b5e7bc109c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 21:53:34.670077 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.670032    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c8a580c9-abfb-4255-8094-60b5e7bc109c" (UID: "c8a580c9-abfb-4255-8094-60b5e7bc109c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 21:53:34.672062 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.672036    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a580c9-abfb-4255-8094-60b5e7bc109c-kube-api-access-bl2c7" (OuterVolumeSpecName: "kube-api-access-bl2c7") pod "c8a580c9-abfb-4255-8094-60b5e7bc109c" (UID: "c8a580c9-abfb-4255-8094-60b5e7bc109c"). InnerVolumeSpecName "kube-api-access-bl2c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 21:53:34.672252 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.672221    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c8a580c9-abfb-4255-8094-60b5e7bc109c" (UID: "c8a580c9-abfb-4255-8094-60b5e7bc109c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 21:53:34.672294 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.672265    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c8a580c9-abfb-4255-8094-60b5e7bc109c" (UID: "c8a580c9-abfb-4255-8094-60b5e7bc109c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 21:53:34.770362 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.770327    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-config\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:53:34.770362 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.770355    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:53:34.770362 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.770366    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-serving-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:53:34.770588 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.770375    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl2c7\" (UniqueName: \"kubernetes.io/projected/c8a580c9-abfb-4255-8094-60b5e7bc109c-kube-api-access-bl2c7\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:53:34.770588 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.770390    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-trusted-ca-bundle\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:53:34.770588 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.770401    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c8a580c9-abfb-4255-8094-60b5e7bc109c-oauth-serving-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:53:34.770588 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.770409    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c8a580c9-abfb-4255-8094-60b5e7bc109c-console-oauth-config\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:53:34.953919 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.953881    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9d597dcfc-n9whk"]
Jul 23 21:53:34.958299 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:34.958270    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9d597dcfc-n9whk"]
Jul 23 21:53:35.111483 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:35.111450    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8a580c9-abfb-4255-8094-60b5e7bc109c" path="/var/lib/kubelet/pods/c8a580c9-abfb-4255-8094-60b5e7bc109c/volumes"
Jul 23 21:53:52.029655 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:52.029616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:52.131329 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:52.131295    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:53:59.827764 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:59.827720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:53:59.847840 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:53:59.847811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:00.739535 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:00.739507    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:03.193663 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:03.193629    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:54:10.018160 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:10.018118    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:54:12.486295 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:12.486256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:54:13.046483 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:13.046432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:54:17.867273 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:17.867235    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 21:54:17.867713 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:17.867556    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="prometheus" containerID="cri-o://21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541" gracePeriod=600
Jul 23 21:54:17.867713 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:17.867605    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="thanos-sidecar" containerID="cri-o://38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476" gracePeriod=600
Jul 23 21:54:17.867713 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:17.867625    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="config-reloader" containerID="cri-o://e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7" gracePeriod=600
Jul 23 21:54:17.867713 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:17.867676    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy" containerID="cri-o://225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b" gracePeriod=600
Jul 23 21:54:17.867948 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:17.867632    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy-web" containerID="cri-o://2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86" gracePeriod=600
Jul 23 21:54:17.867948 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:17.867605    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy-thanos" containerID="cri-o://67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64" gracePeriod=600
Jul 23 21:54:18.116945 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.116920    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:18.211035 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211006    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-2\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211192 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-kubelet-serving-ca-bundle\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211192 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211081    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-tls\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211192 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211111    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-metrics-client-ca\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211192 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211136    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-metrics-client-certs\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211192 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211183    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-config-out\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211434 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211213    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9mn9\" (UniqueName: \"kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-kube-api-access-d9mn9\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211434 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211241    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-tls-assets\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211434 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211266    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211434 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-config\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211434 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211316    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-web-config\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211434 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211343    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-0\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211434 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211390    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-serving-certs-ca-bundle\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211434 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211415    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-grpc-tls\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211857 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211448    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-db\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211857 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-1\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211857 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-trusted-ca-bundle\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211857 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211508    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 21:54:18.211857 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211555    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-kube-rbac-proxy\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211857 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211572    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 21:54:18.211857 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211578    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-thanos-prometheus-http-client-file\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211857 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211743    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ea513b0f-adfe-4b97-9c10-917880a20b99\" (UID: \"ea513b0f-adfe-4b97-9c10-917880a20b99\") "
Jul 23 21:54:18.211857 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 21:54:18.212349 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.211918    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 21:54:18.212349 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.212141    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 21:54:18.212349 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.212192    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.212349 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.212215    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.212349 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.212231    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.212349 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.212246    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-configmap-metrics-client-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.213167 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.212972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 21:54:18.214753 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.214722    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 21:54:18.215241 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.215217    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 21:54:18.215409 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.215379    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-kube-api-access-d9mn9" (OuterVolumeSpecName: "kube-api-access-d9mn9") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "kube-api-access-d9mn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 21:54:18.215409 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.215393    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 21:54:18.215531 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.215462    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 21:54:18.215584 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.215563    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 21:54:18.215779 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.215757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-config-out" (OuterVolumeSpecName: "config-out") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 21:54:18.215843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.215818    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 21:54:18.216193 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.216169    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 21:54:18.216242 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.216201    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 21:54:18.216439 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.216419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-config" (OuterVolumeSpecName: "config") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 21:54:18.216854 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.216835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 21:54:18.216985 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.216960    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 21:54:18.227180 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.227158    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-web-config" (OuterVolumeSpecName: "web-config") pod "ea513b0f-adfe-4b97-9c10-917880a20b99" (UID: "ea513b0f-adfe-4b97-9c10-917880a20b99"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 21:54:18.313305 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313268    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-config-out\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313305 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313302    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9mn9\" (UniqueName: \"kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-kube-api-access-d9mn9\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313315    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea513b0f-adfe-4b97-9c10-917880a20b99-tls-assets\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313329    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313340    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-config\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313377    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-web-config\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313390    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313400    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-grpc-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313411    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-db\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313423    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313438    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea513b0f-adfe-4b97-9c10-917880a20b99-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313450    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-kube-rbac-proxy\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313463    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313477    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313492    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.313518 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.313508    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ea513b0f-adfe-4b97-9c10-917880a20b99-secret-metrics-client-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 21:54:18.789554 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789522    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerID="67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64" exitCode=0
Jul 23 21:54:18.789554 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789547    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerID="225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b" exitCode=0
Jul 23 21:54:18.789554 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789553    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerID="2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86" exitCode=0
Jul 23 21:54:18.789554 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789559    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerID="38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476" exitCode=0
Jul 23 21:54:18.789554 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789564    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerID="e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7" exitCode=0
Jul 23 21:54:18.789843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789569    2579 generic.go:356] "Generic (PLEG): container finished" podID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerID="21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541" exitCode=0
Jul 23 21:54:18.789843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerDied","Data":"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"}
Jul 23 21:54:18.789843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789610    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:18.789843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerDied","Data":"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"}
Jul 23 21:54:18.789843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerDied","Data":"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"}
Jul 23 21:54:18.789843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerDied","Data":"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"}
Jul 23 21:54:18.789843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerDied","Data":"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"}
Jul 23 21:54:18.789843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789736    2579 scope.go:117] "RemoveContainer" containerID="67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"
Jul 23 21:54:18.789843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerDied","Data":"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"}
Jul 23 21:54:18.789843 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.789777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ea513b0f-adfe-4b97-9c10-917880a20b99","Type":"ContainerDied","Data":"c42167bdd5b8d4aaa635d783756854c37950333aa16af8d9e10e6ebda734c2e5"}
Jul 23 21:54:18.797932 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.797906    2579 scope.go:117] "RemoveContainer" containerID="225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"
Jul 23 21:54:18.805190 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.805168    2579 scope.go:117] "RemoveContainer" containerID="2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"
Jul 23 21:54:18.812670 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.812650    2579 scope.go:117] "RemoveContainer" containerID="38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"
Jul 23 21:54:18.813007 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.812986    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 21:54:18.819967 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.819940    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 21:54:18.821019 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.821004    2579 scope.go:117] "RemoveContainer" containerID="e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"
Jul 23 21:54:18.828139 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.828113    2579 scope.go:117] "RemoveContainer" containerID="21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"
Jul 23 21:54:18.835420 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.835401    2579 scope.go:117] "RemoveContainer" containerID="5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"
Jul 23 21:54:18.842562 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.842543    2579 scope.go:117] "RemoveContainer" containerID="67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"
Jul 23 21:54:18.842845 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:54:18.842825    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": container with ID starting with 67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64 not found: ID does not exist" containerID="67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"
Jul 23 21:54:18.842947 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.842872    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"} err="failed to get container status \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": rpc error: code = NotFound desc = could not find container \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": container with ID starting with 67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64 not found: ID does not exist"
Jul 23 21:54:18.842947 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.842893    2579 scope.go:117] "RemoveContainer" containerID="225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"
Jul 23 21:54:18.843135 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:54:18.843114    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": container with ID starting with 225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b not found: ID does not exist" containerID="225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"
Jul 23 21:54:18.843199 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.843146    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"} err="failed to get container status \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": rpc error: code = NotFound desc = could not find container \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": container with ID starting with 225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b not found: ID does not exist"
Jul 23 21:54:18.843199 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.843170    2579 scope.go:117] "RemoveContainer" containerID="2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"
Jul 23 21:54:18.843383 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:54:18.843364    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": container with ID starting with 2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86 not found: ID does not exist" containerID="2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"
Jul 23 21:54:18.843424 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.843388    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"} err="failed to get container status \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": rpc error: code = NotFound desc = could not find container \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": container with ID starting with 2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86 not found: ID does not exist"
Jul 23 21:54:18.843424 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.843405    2579 scope.go:117] "RemoveContainer" containerID="38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"
Jul 23 21:54:18.843650 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:54:18.843634    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": container with ID starting with 38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476 not found: ID does not exist" containerID="38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"
Jul 23 21:54:18.843711 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.843656    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"} err="failed to get container status \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": rpc error: code = NotFound desc = could not find container \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": container with ID starting with 38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476 not found: ID does not exist"
Jul 23 21:54:18.843711 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.843676    2579 scope.go:117] "RemoveContainer" containerID="e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"
Jul 23 21:54:18.843909 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:54:18.843889    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": container with ID starting with e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7 not found: ID does not exist" containerID="e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"
Jul 23 21:54:18.843959 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.843915    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"} err="failed to get container status \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": rpc error: code = NotFound desc = could not find container \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": container with ID starting with e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7 not found: ID does not exist"
Jul 23 21:54:18.843959 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.843938    2579 scope.go:117] "RemoveContainer" containerID="21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"
Jul 23 21:54:18.844209 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:54:18.844184    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": container with ID starting with 21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541 not found: ID does not exist" containerID="21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"
Jul 23 21:54:18.844249 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.844214    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"} err="failed to get container status \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": rpc error: code = NotFound desc = could not find container \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": container with ID starting with 21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541 not found: ID does not exist"
Jul 23 21:54:18.844249 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.844229    2579 scope.go:117] "RemoveContainer" containerID="5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"
Jul 23 21:54:18.844475 ip-10-0-135-91 kubenswrapper[2579]: E0723 21:54:18.844460    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": container with ID starting with 5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f not found: ID does not exist" containerID="5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"
Jul 23 21:54:18.844514 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.844479    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"} err="failed to get container status \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": rpc error: code = NotFound desc = could not find container \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": container with ID starting with 5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f not found: ID does not exist"
Jul 23 21:54:18.844514 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.844491    2579 scope.go:117] "RemoveContainer" containerID="67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"
Jul 23 21:54:18.844696 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.844675    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"} err="failed to get container status \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": rpc error: code = NotFound desc = could not find container \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": container with ID starting with 67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64 not found: ID does not exist"
Jul 23 21:54:18.844739 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.844696    2579 scope.go:117] "RemoveContainer" containerID="225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"
Jul 23 21:54:18.844932 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.844916    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"} err="failed to get container status \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": rpc error: code = NotFound desc = could not find container \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": container with ID starting with 225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b not found: ID does not exist"
Jul 23 21:54:18.844932 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.844930    2579 scope.go:117] "RemoveContainer" containerID="2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"
Jul 23 21:54:18.845130 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.845116    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"} err="failed to get container status \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": rpc error: code = NotFound desc = could not find container \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": container with ID starting with 2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86 not found: ID does not exist"
Jul 23 21:54:18.845130 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.845129    2579 scope.go:117] "RemoveContainer" containerID="38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"
Jul 23 21:54:18.845346 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.845326    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"} err="failed to get container status \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": rpc error: code = NotFound desc = could not find container \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": container with ID starting with 38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476 not found: ID does not exist"
Jul 23 21:54:18.845426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.845348    2579 scope.go:117] "RemoveContainer" containerID="e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"
Jul 23 21:54:18.845591 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.845573    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"} err="failed to get container status \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": rpc error: code = NotFound desc = could not find container \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": container with ID starting with e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7 not found: ID does not exist"
Jul 23 21:54:18.845639 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.845591    2579 scope.go:117] "RemoveContainer" containerID="21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"
Jul 23 21:54:18.845823 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.845807    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"} err="failed to get container status \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": rpc error: code = NotFound desc = could not find container \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": container with ID starting with 21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541 not found: ID does not exist"
Jul 23 21:54:18.845900 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.845824    2579 scope.go:117] "RemoveContainer" containerID="5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"
Jul 23 21:54:18.846052 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.846029    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"} err="failed to get container status \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": rpc error: code = NotFound desc = could not find container \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": container with ID starting with 5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f not found: ID does not exist"
Jul 23 21:54:18.846102 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.846054    2579 scope.go:117] "RemoveContainer" containerID="67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"
Jul 23 21:54:18.846528 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.846502    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"} err="failed to get container status \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": rpc error: code = NotFound desc = could not find container \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": container with ID starting with 67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64 not found: ID does not exist"
Jul 23 21:54:18.846528 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.846529    2579 scope.go:117] "RemoveContainer" containerID="225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"
Jul 23 21:54:18.846972 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.846872    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"} err="failed to get container status \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": rpc error: code = NotFound desc = could not find container \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": container with ID starting with 225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b not found: ID does not exist"
Jul 23 21:54:18.846972 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.846900    2579 scope.go:117] "RemoveContainer" containerID="2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"
Jul 23 21:54:18.847177 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.847144    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"} err="failed to get container status \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": rpc error: code = NotFound desc = could not find container \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": container with ID starting with 2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86 not found: ID does not exist"
Jul 23 21:54:18.847232 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.847181    2579 scope.go:117] "RemoveContainer" containerID="38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"
Jul 23 21:54:18.847476 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.847453    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"} err="failed to get container status \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": rpc error: code = NotFound desc = could not find container \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": container with ID starting with 38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476 not found: ID does not exist"
Jul 23 21:54:18.847476 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.847472    2579 scope.go:117] "RemoveContainer" containerID="e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"
Jul 23 21:54:18.847728 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.847708    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"} err="failed to get container status \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": rpc error: code = NotFound desc = could not find container \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": container with ID starting with e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7 not found: ID does not exist"
Jul 23 21:54:18.847805 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.847739    2579 scope.go:117] "RemoveContainer" containerID="21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"
Jul 23 21:54:18.848063 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848043    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 21:54:18.848140 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848059    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"} err="failed to get container status \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": rpc error: code = NotFound desc = could not find container \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": container with ID starting with 21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541 not found: ID does not exist"
Jul 23 21:54:18.848140 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848077    2579 scope.go:117] "RemoveContainer" containerID="5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"
Jul 23 21:54:18.848336 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848318    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy"
Jul 23 21:54:18.848413 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848328    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"} err="failed to get container status \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": rpc error: code = NotFound desc = could not find container \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": container with ID starting with 5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f not found: ID does not exist"
Jul 23 21:54:18.848413 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848351    2579 scope.go:117] "RemoveContainer" containerID="67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"
Jul 23 21:54:18.848413 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848339    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848427    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a580c9-abfb-4255-8094-60b5e7bc109c" containerName="console"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848439    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a580c9-abfb-4255-8094-60b5e7bc109c" containerName="console"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848447    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="config-reloader"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848452    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="config-reloader"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848457    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="thanos-sidecar"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848462    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="thanos-sidecar"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848468    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy-web"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848473    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy-web"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848479    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy-thanos"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848483    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy-thanos"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848490    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="init-config-reloader"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848495    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="init-config-reloader"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848500    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="prometheus"
Jul 23 21:54:18.848532 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848505    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="prometheus"
Jul 23 21:54:18.849021 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848574    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a580c9-abfb-4255-8094-60b5e7bc109c" containerName="console"
Jul 23 21:54:18.849021 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848584    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="thanos-sidecar"
Jul 23 21:54:18.849021 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848590    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy-web"
Jul 23 21:54:18.849021 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848597    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy"
Jul 23 21:54:18.849021 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848606    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="config-reloader"
Jul 23 21:54:18.849021 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848612    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="prometheus"
Jul 23 21:54:18.849021 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848618    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" containerName="kube-rbac-proxy-thanos"
Jul 23 21:54:18.849021 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848606    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"} err="failed to get container status \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": rpc error: code = NotFound desc = could not find container \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": container with ID starting with 67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64 not found: ID does not exist"
Jul 23 21:54:18.849021 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848633    2579 scope.go:117] "RemoveContainer" containerID="225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"
Jul 23 21:54:18.849021 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848882    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"} err="failed to get container status \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": rpc error: code = NotFound desc = could not find container \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": container with ID starting with 225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b not found: ID does not exist"
Jul 23 21:54:18.849021 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.848898    2579 scope.go:117] "RemoveContainer" containerID="2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"
Jul 23 21:54:18.849426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.849123    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"} err="failed to get container status \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": rpc error: code = NotFound desc = could not find container \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": container with ID starting with 2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86 not found: ID does not exist"
Jul 23 21:54:18.849426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.849148    2579 scope.go:117] "RemoveContainer" containerID="38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"
Jul 23 21:54:18.849426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.849372    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"} err="failed to get container status \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": rpc error: code = NotFound desc = could not find container \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": container with ID starting with 38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476 not found: ID does not exist"
Jul 23 21:54:18.849426 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.849391    2579 scope.go:117] "RemoveContainer" containerID="e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"
Jul 23 21:54:18.849627 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.849603    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"} err="failed to get container status \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": rpc error: code = NotFound desc = could not find container \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": container with ID starting with e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7 not found: ID does not exist"
Jul 23 21:54:18.849662 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.849629    2579 scope.go:117] "RemoveContainer" containerID="21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"
Jul 23 21:54:18.849839 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.849822    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"} err="failed to get container status \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": rpc error: code = NotFound desc = could not find container \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": container with ID starting with 21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541 not found: ID does not exist"
Jul 23 21:54:18.849914 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.849838    2579 scope.go:117] "RemoveContainer" containerID="5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"
Jul 23 21:54:18.850059 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.850043    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"} err="failed to get container status \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": rpc error: code = NotFound desc = could not find container \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": container with ID starting with 5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f not found: ID does not exist"
Jul 23 21:54:18.850103 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.850060    2579 scope.go:117] "RemoveContainer" containerID="67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"
Jul 23 21:54:18.850276 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.850245    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"} err="failed to get container status \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": rpc error: code = NotFound desc = could not find container \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": container with ID starting with 67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64 not found: ID does not exist"
Jul 23 21:54:18.850319 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.850278    2579 scope.go:117] "RemoveContainer" containerID="225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"
Jul 23 21:54:18.850520 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.850502    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"} err="failed to get container status \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": rpc error: code = NotFound desc = could not find container \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": container with ID starting with 225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b not found: ID does not exist"
Jul 23 21:54:18.850562 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.850521    2579 scope.go:117] "RemoveContainer" containerID="2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"
Jul 23 21:54:18.850734 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.850719    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"} err="failed to get container status \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": rpc error: code = NotFound desc = could not find container \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": container with ID starting with 2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86 not found: ID does not exist"
Jul 23 21:54:18.850734 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.850734    2579 scope.go:117] "RemoveContainer" containerID="38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"
Jul 23 21:54:18.850975 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.850957    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"} err="failed to get container status \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": rpc error: code = NotFound desc = could not find container \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": container with ID starting with 38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476 not found: ID does not exist"
Jul 23 21:54:18.851037 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.850975    2579 scope.go:117] "RemoveContainer" containerID="e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"
Jul 23 21:54:18.851192 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.851175    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"} err="failed to get container status \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": rpc error: code = NotFound desc = could not find container \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": container with ID starting with e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7 not found: ID does not exist"
Jul 23 21:54:18.851244 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.851192    2579 scope.go:117] "RemoveContainer" containerID="21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"
Jul 23 21:54:18.851421 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.851404    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"} err="failed to get container status \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": rpc error: code = NotFound desc = could not find container \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": container with ID starting with 21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541 not found: ID does not exist"
Jul 23 21:54:18.851472 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.851421    2579 scope.go:117] "RemoveContainer" containerID="5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"
Jul 23 21:54:18.851656 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.851632    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"} err="failed to get container status \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": rpc error: code = NotFound desc = could not find container \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": container with ID starting with 5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f not found: ID does not exist"
Jul 23 21:54:18.851693 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.851656    2579 scope.go:117] "RemoveContainer" containerID="67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"
Jul 23 21:54:18.851873 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.851844    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64"} err="failed to get container status \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": rpc error: code = NotFound desc = could not find container \"67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64\": container with ID starting with 67a735166758a287f4bfbc45764122429f0fbeef5ef2aaae892e884f20121d64 not found: ID does not exist"
Jul 23 21:54:18.851939 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.851879    2579 scope.go:117] "RemoveContainer" containerID="225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"
Jul 23 21:54:18.852111 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.852086    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b"} err="failed to get container status \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": rpc error: code = NotFound desc = could not find container \"225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b\": container with ID starting with 225fa91cec3c9bc3fb7d5663a6625a24594251828f2628855d3f733853b4686b not found: ID does not exist"
Jul 23 21:54:18.852157 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.852111    2579 scope.go:117] "RemoveContainer" containerID="2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"
Jul 23 21:54:18.852322 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.852302    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86"} err="failed to get container status \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": rpc error: code = NotFound desc = could not find container \"2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86\": container with ID starting with 2fc0ed7b18b1055fb06de7ece7923cc4d5ffeb00c9bed12cab9f68be27052b86 not found: ID does not exist"
Jul 23 21:54:18.852368 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.852323    2579 scope.go:117] "RemoveContainer" containerID="38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"
Jul 23 21:54:18.852528 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.852511    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476"} err="failed to get container status \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": rpc error: code = NotFound desc = could not find container \"38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476\": container with ID starting with 38d8c4eb3d8e7b5d0d451a887cc86cdcceb508bc3921e9e1388d25be81dbe476 not found: ID does not exist"
Jul 23 21:54:18.852571 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.852528    2579 scope.go:117] "RemoveContainer" containerID="e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"
Jul 23 21:54:18.852736 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.852718    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7"} err="failed to get container status \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": rpc error: code = NotFound desc = could not find container \"e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7\": container with ID starting with e25bad5827fb58baace057ceda648f50c76d946ed8131f0019b246f113612ad7 not found: ID does not exist"
Jul 23 21:54:18.852785 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.852736    2579 scope.go:117] "RemoveContainer" containerID="21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"
Jul 23 21:54:18.852958 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.852944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:18.853006 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.852994    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541"} err="failed to get container status \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": rpc error: code = NotFound desc = could not find container \"21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541\": container with ID starting with 21b61912f137135ab375aad055b1cd1f404c08ecc5b0be1c09a881e829685541 not found: ID does not exist"
Jul 23 21:54:18.853042 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.853010    2579 scope.go:117] "RemoveContainer" containerID="5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"
Jul 23 21:54:18.853294 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.853259    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f"} err="failed to get container status \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": rpc error: code = NotFound desc = could not find container \"5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f\": container with ID starting with 5859592fdc12b4f86fb1367b96b3962847ae1bc47736f9e4fa324fbb05300e5f not found: ID does not exist"
Jul 23 21:54:18.855613 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.855587    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 21:54:18.855726 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.855633    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-akrnbk3qbtfun\""
Jul 23 21:54:18.855792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.855730    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 21:54:18.855792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.855748    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 21:54:18.856067 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.856051    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cxkvl\""
Jul 23 21:54:18.856156 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.856092    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 21:54:18.856213 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.856163    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 21:54:18.856213 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.856176    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 21:54:18.856306 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.856224    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 21:54:18.856398 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.856373    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 21:54:18.856456 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.856415    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 21:54:18.856663 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.856646    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 21:54:18.856778 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.856763    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 21:54:18.856836 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.856823    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 21:54:18.857258 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.857217    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 21:54:18.859046 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.859025    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 21:54:18.861596 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.861579    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 21:54:18.870838 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:18.870813    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 21:54:19.019331 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3222819f-a6a5-42dd-b48f-adb8bd198885-config-out\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019508 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019508 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019508 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019508 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019508 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019508 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019508 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019718 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019718 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019718 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3222819f-a6a5-42dd-b48f-adb8bd198885-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019718 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-web-config\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019718 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019718 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019718 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-config\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019960 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019722    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019960 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019960 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019960 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.019960 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.019836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2l8g\" (UniqueName: \"kubernetes.io/projected/3222819f-a6a5-42dd-b48f-adb8bd198885-kube-api-access-c2l8g\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.114378 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.110926    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea513b0f-adfe-4b97-9c10-917880a20b99" path="/var/lib/kubelet/pods/ea513b0f-adfe-4b97-9c10-917880a20b99/volumes"
Jul 23 21:54:19.121004 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.120954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121075 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121075 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-config\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121075 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121180 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121269 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121326 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121382 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2l8g\" (UniqueName: \"kubernetes.io/projected/3222819f-a6a5-42dd-b48f-adb8bd198885-kube-api-access-c2l8g\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121382 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3222819f-a6a5-42dd-b48f-adb8bd198885-config-out\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121485 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121485 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121705 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121812 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.121812 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.122111 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.121838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.122111 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.122013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.122111 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.122038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.122111 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.122080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.122328 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.122120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.122328 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.122145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.122328 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.122177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3222819f-a6a5-42dd-b48f-adb8bd198885-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.122328 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.122211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-web-config\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.122522 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.122334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.122694 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.122670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.124779 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.124530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-config\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.124779 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.124602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.125014 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.124830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.125014 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.125002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.125121 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.125056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.125880 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.125397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3222819f-a6a5-42dd-b48f-adb8bd198885-config-out\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.125880 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.125472    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-web-config\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.125880 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.125545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.125880 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.125799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.126125 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.125964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.126834 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.126802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.127596 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.127571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.128126 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.128099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3222819f-a6a5-42dd-b48f-adb8bd198885-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.128232 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.128141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3222819f-a6a5-42dd-b48f-adb8bd198885-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.128637 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.128621    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3222819f-a6a5-42dd-b48f-adb8bd198885-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.129845 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.129827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2l8g\" (UniqueName: \"kubernetes.io/projected/3222819f-a6a5-42dd-b48f-adb8bd198885-kube-api-access-c2l8g\") pod \"prometheus-k8s-0\" (UID: \"3222819f-a6a5-42dd-b48f-adb8bd198885\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.163639 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.163568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:19.303111 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.303074    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 21:54:19.306037 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:54:19.306009    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3222819f_a6a5_42dd_b48f_adb8bd198885.slice/crio-ba47653f55e696f7304c7a44a7a826af6b01f76cac808ea516839c4bcf87147e WatchSource:0}: Error finding container ba47653f55e696f7304c7a44a7a826af6b01f76cac808ea516839c4bcf87147e: Status 404 returned error can't find the container with id ba47653f55e696f7304c7a44a7a826af6b01f76cac808ea516839c4bcf87147e
Jul 23 21:54:19.794814 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.794776    2579 generic.go:356] "Generic (PLEG): container finished" podID="3222819f-a6a5-42dd-b48f-adb8bd198885" containerID="5409d3d18c44dcdcfbaca9c25616cf1d685237a94bc2d5b91a6a05cc26864a81" exitCode=0
Jul 23 21:54:19.795016 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.794881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3222819f-a6a5-42dd-b48f-adb8bd198885","Type":"ContainerDied","Data":"5409d3d18c44dcdcfbaca9c25616cf1d685237a94bc2d5b91a6a05cc26864a81"}
Jul 23 21:54:19.795016 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:19.794964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3222819f-a6a5-42dd-b48f-adb8bd198885","Type":"ContainerStarted","Data":"ba47653f55e696f7304c7a44a7a826af6b01f76cac808ea516839c4bcf87147e"}
Jul 23 21:54:20.803389 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:20.803345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3222819f-a6a5-42dd-b48f-adb8bd198885","Type":"ContainerStarted","Data":"bd3ab53529d2be7bc044be20c086338db116a8d52b7340e89aff9ff35859b2d6"}
Jul 23 21:54:20.803805 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:20.803410    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3222819f-a6a5-42dd-b48f-adb8bd198885","Type":"ContainerStarted","Data":"504fec0be0b261f475a3384578a2cce7e1bb0db1097598fa77cd8f3eb3d24001"}
Jul 23 21:54:20.803805 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:20.803437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3222819f-a6a5-42dd-b48f-adb8bd198885","Type":"ContainerStarted","Data":"a9698e11c0b3548db259c371595eeed171f5be8344c81863147cdc085767a3e8"}
Jul 23 21:54:20.803805 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:20.803466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3222819f-a6a5-42dd-b48f-adb8bd198885","Type":"ContainerStarted","Data":"0500d3eb2c538a46bbd6de8626c2c58a95ef1af00343a0a1456e273baf5e4101"}
Jul 23 21:54:20.803805 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:20.803491    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3222819f-a6a5-42dd-b48f-adb8bd198885","Type":"ContainerStarted","Data":"e155521b785c0137b6009b18273db2c2a21c3600c0080fc5f76bc281d49fd1fd"}
Jul 23 21:54:20.803805 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:20.803513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3222819f-a6a5-42dd-b48f-adb8bd198885","Type":"ContainerStarted","Data":"736d43dde09832745d227a9138d8befeba01c983224dbcd3f8f75715d530cd88"}
Jul 23 21:54:20.880695 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:20.880641    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.85946847 podStartE2EDuration="2.880622336s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.153866ms" isStatefulPod=false podCreationTimestamp="2026-07-23 21:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:54:20.879001681 +0000 UTC m=+156.269394299" watchObservedRunningTime="2026-07-23 21:54:20.880622336 +0000 UTC m=+156.271014952"
Jul 23 21:54:21.586109 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:21.586071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:54:21.687139 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:21.687104    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:54:24.163903 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:24.163847    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:54:32.826492 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:32.826453    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:54:32.928028 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:32.927980    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:54:32.984914 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:54:32.984879    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:04.379127 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:04.379087    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:17.952601 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:17.952559    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:18.034494 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:18.034455    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:19.164303 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:19.164269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:55:19.180193 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:19.180168    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:55:20.018959 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:20.018931    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 21:55:27.754048 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:27.754005    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:27.813058 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:27.813017    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:33.198065 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:33.198032    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:33.298751 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:33.298719    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:33.399964 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:33.399931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:33.933855 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:33.933822    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:34.034083 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:34.034047    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:39.743292 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:39.743251    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:39.814341 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:39.814307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:44.143651 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:44.143602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:44.244133 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:44.244096    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:44.345064 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:44.345027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:57.203639 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:57.203604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:57.303717 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:57.303683    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:55:57.401327 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:55:57.401289    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:28.692268 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:28.692235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:28.789261 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:28.789229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:44.976003 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:44.975973    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 21:56:44.981841 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:44.981815    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 21:56:44.984261 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:44.984239    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 21:56:50.453121 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:50.453077    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:51.682337 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:51.682299    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:54.068932 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:54.068899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:54.164969 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:54.164915    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:56.407457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:56.407408    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:56.512344 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:56.512295    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:56.757498 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:56.757460    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:56.888408 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:56.888372    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:58.395848 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:58.395811    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:56:58.787650 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:56:58.787610    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:01.063771 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:01.063737    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:01.165109 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:01.165072    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:04.031684 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:04.031644    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:04.132102 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:04.132056    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:07.062606 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:07.062569    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:12.665162 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:12.665129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:12.761911 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:12.761855    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:19.130972 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:19.130926    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:19.231268 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:19.231229    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:20.823931 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:20.823889    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:26.296168 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:26.296115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:26.353035 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:26.352984    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:29.053010 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.052977    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:29.097002 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.096965    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"]
Jul 23 21:57:29.099022 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.099003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.102323 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.102297    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-68fvx\""
Jul 23 21:57:29.102472 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.102326    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 23 21:57:29.102472 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.102326    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 23 21:57:29.102472 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.102327    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 23 21:57:29.102472 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.102297    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 23 21:57:29.102472 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.102401    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 23 21:57:29.110336 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.110314    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"]
Jul 23 21:57:29.133301 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.133272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:29.237262 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.237223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-cert\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.237262 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.237262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqlnt\" (UniqueName: \"kubernetes.io/projected/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-kube-api-access-gqlnt\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.237457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.237287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-metrics-cert\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.237457 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.237358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-manager-config\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.338448 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.338369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-cert\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.338448 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.338407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqlnt\" (UniqueName: \"kubernetes.io/projected/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-kube-api-access-gqlnt\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.338448 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.338435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-metrics-cert\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.338647 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.338474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-manager-config\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.339279 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.339203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-manager-config\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.340968 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.340945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-cert\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.341114 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.341093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-metrics-cert\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.352586 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.352558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqlnt\" (UniqueName: \"kubernetes.io/projected/81b8750e-08ea-4ae7-ac3b-effa41bc7fe4-kube-api-access-gqlnt\") pod \"lws-controller-manager-7575c4d6b8-792bd\" (UID: \"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4\") " pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.408445 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.408399    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:29.535963 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.535931    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"]
Jul 23 21:57:29.539211 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:57:29.539176    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b8750e_08ea_4ae7_ac3b_effa41bc7fe4.slice/crio-738299afeec414b05182967daa3bf95160f2b83c2e611983d4698b5531ee7bcf WatchSource:0}: Error finding container 738299afeec414b05182967daa3bf95160f2b83c2e611983d4698b5531ee7bcf: Status 404 returned error can't find the container with id 738299afeec414b05182967daa3bf95160f2b83c2e611983d4698b5531ee7bcf
Jul 23 21:57:29.541008 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:29.540987    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 21:57:30.440698 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:30.440663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd" event={"ID":"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4","Type":"ContainerStarted","Data":"738299afeec414b05182967daa3bf95160f2b83c2e611983d4698b5531ee7bcf"}
Jul 23 21:57:32.448963 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:32.448922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd" event={"ID":"81b8750e-08ea-4ae7-ac3b-effa41bc7fe4","Type":"ContainerStarted","Data":"ac40bb029ce2f180d4a84a7c02573bf00b31feec32762e4a8e7139f6b34aee7b"}
Jul 23 21:57:32.449394 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:32.449077    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:32.469269 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:32.469214    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd" podStartSLOduration=1.121655095 podStartE2EDuration="3.469195078s" totalImagesPullingTime="2.347539983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:57:32.468480207 +0000 UTC m=+347.858872826" watchObservedRunningTime="2026-07-23 21:57:32.469195078 +0000 UTC m=+347.859587697"
Jul 23 21:57:32.881635 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:32.881599    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:32.975762 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:32.975730    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:33.496648 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.496610    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:33.737171 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.737134    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:33.834189 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.834097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:33.880152 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.880120    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"]
Jul 23 21:57:33.883143 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.883125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:33.889647 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.889560    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 21:57:33.889768 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.889747    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 21:57:33.890401 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.890385    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 21:57:33.890478 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.890449    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 21:57:33.891060 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.891045    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qjdhj\""
Jul 23 21:57:33.907839 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.907804    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"]
Jul 23 21:57:33.980380 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.980330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc156a06-fd78-4756-9283-569e236dcc64-webhook-cert\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:33.980380 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.980385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc156a06-fd78-4756-9283-569e236dcc64-manifests\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:33.980625 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.980410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bc156a06-fd78-4756-9283-569e236dcc64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:33.980625 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.980463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxm7r\" (UniqueName: \"kubernetes.io/projected/bc156a06-fd78-4756-9283-569e236dcc64-kube-api-access-nxm7r\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:33.980625 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:33.980489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc156a06-fd78-4756-9283-569e236dcc64-tmp\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.073345 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.073291    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:34.081358 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.081331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc156a06-fd78-4756-9283-569e236dcc64-webhook-cert\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.081493 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.081366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc156a06-fd78-4756-9283-569e236dcc64-manifests\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.081493 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.081387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bc156a06-fd78-4756-9283-569e236dcc64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.081591 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.081527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxm7r\" (UniqueName: \"kubernetes.io/projected/bc156a06-fd78-4756-9283-569e236dcc64-kube-api-access-nxm7r\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.081591 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.081564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc156a06-fd78-4756-9283-569e236dcc64-tmp\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.081815 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.081793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bc156a06-fd78-4756-9283-569e236dcc64-manifests\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.083918 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.083889    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc156a06-fd78-4756-9283-569e236dcc64-tmp\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.083996 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.083979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bc156a06-fd78-4756-9283-569e236dcc64-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.084118 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.084103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bc156a06-fd78-4756-9283-569e236dcc64-webhook-cert\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.090327 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.090274    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxm7r\" (UniqueName: \"kubernetes.io/projected/bc156a06-fd78-4756-9283-569e236dcc64-kube-api-access-nxm7r\") pod \"opendatahub-operator-controller-manager-6864c64679-4cjcz\" (UID: \"bc156a06-fd78-4756-9283-569e236dcc64\") " pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.173915 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.173846    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:34.193184 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.193147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:34.272703 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.272672    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:34.330497 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.330449    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"]
Jul 23 21:57:34.332852 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:57:34.332823    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc156a06_fd78_4756_9283_569e236dcc64.slice/crio-8af734e5cac68d8e451054a55714ddeebc6117edddc90a3f60248490ab22a5d1 WatchSource:0}: Error finding container 8af734e5cac68d8e451054a55714ddeebc6117edddc90a3f60248490ab22a5d1: Status 404 returned error can't find the container with id 8af734e5cac68d8e451054a55714ddeebc6117edddc90a3f60248490ab22a5d1
Jul 23 21:57:34.342455 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.342393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:34.426075 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.426047    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:34.459168 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.459136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz" event={"ID":"bc156a06-fd78-4756-9283-569e236dcc64","Type":"ContainerStarted","Data":"8af734e5cac68d8e451054a55714ddeebc6117edddc90a3f60248490ab22a5d1"}
Jul 23 21:57:34.525936 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.525899    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:34.622607 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:34.622515    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:37.472721 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:37.472681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz" event={"ID":"bc156a06-fd78-4756-9283-569e236dcc64","Type":"ContainerStarted","Data":"8250b04fc0adf87a696ea46bfda28f0de678196a463dca2b5a7caa14c083ddad"}
Jul 23 21:57:38.478248 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:38.478212    2579 generic.go:356] "Generic (PLEG): container finished" podID="bc156a06-fd78-4756-9283-569e236dcc64" containerID="8250b04fc0adf87a696ea46bfda28f0de678196a463dca2b5a7caa14c083ddad" exitCode=0
Jul 23 21:57:38.478629 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:38.478298    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz" event={"ID":"bc156a06-fd78-4756-9283-569e236dcc64","Type":"ContainerDied","Data":"8250b04fc0adf87a696ea46bfda28f0de678196a463dca2b5a7caa14c083ddad"}
Jul 23 21:57:39.485223 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:39.485188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz" event={"ID":"bc156a06-fd78-4756-9283-569e236dcc64","Type":"ContainerStarted","Data":"f83b986d01dc715e76defc71caa67c76f7606eefaf88df590c9b394bba5f35f4"}
Jul 23 21:57:39.485614 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:39.485328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:39.507450 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:39.507381    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz" podStartSLOduration=2.177233808 podStartE2EDuration="6.507359233s" totalImagesPullingTime="3.270032348s" totalInitContainerRuntime="1.060093077s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:57:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:57:39.506065988 +0000 UTC m=+354.896458605" watchObservedRunningTime="2026-07-23 21:57:39.507359233 +0000 UTC m=+354.897751852"
Jul 23 21:57:43.457091 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:43.457060    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7575c4d6b8-792bd"
Jul 23 21:57:49.759948 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:49.759905    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:50.491941 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:50.491907    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6864c64679-4cjcz"
Jul 23 21:57:53.413755 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:53.413709    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:53.541201 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:53.541164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:59.401897 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.401843    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:59.451978 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.451937    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"]
Jul 23 21:57:59.464923 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.464885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"
Jul 23 21:57:59.468273 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.468237    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r9c92\""
Jul 23 21:57:59.468433 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.468289    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 21:57:59.468433 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.468369    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 23 21:57:59.468433 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.468377    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 23 21:57:59.468565 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.468525    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 21:57:59.470274 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.470253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"]
Jul 23 21:57:59.571741 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.571708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bcc10ec-7c22-44dc-b85a-00b4e3adf48e-tls-certs\") pod \"kube-auth-proxy-5b78d9f867-wc99x\" (UID: \"0bcc10ec-7c22-44dc-b85a-00b4e3adf48e\") " pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"
Jul 23 21:57:59.571741 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.571745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bcc10ec-7c22-44dc-b85a-00b4e3adf48e-tmp\") pod \"kube-auth-proxy-5b78d9f867-wc99x\" (UID: \"0bcc10ec-7c22-44dc-b85a-00b4e3adf48e\") " pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"
Jul 23 21:57:59.571991 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.571772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4xn8\" (UniqueName: \"kubernetes.io/projected/0bcc10ec-7c22-44dc-b85a-00b4e3adf48e-kube-api-access-z4xn8\") pod \"kube-auth-proxy-5b78d9f867-wc99x\" (UID: \"0bcc10ec-7c22-44dc-b85a-00b4e3adf48e\") " pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"
Jul 23 21:57:59.625929 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.625897    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:59.672792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.672703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bcc10ec-7c22-44dc-b85a-00b4e3adf48e-tls-certs\") pod \"kube-auth-proxy-5b78d9f867-wc99x\" (UID: \"0bcc10ec-7c22-44dc-b85a-00b4e3adf48e\") " pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"
Jul 23 21:57:59.672792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.672755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bcc10ec-7c22-44dc-b85a-00b4e3adf48e-tmp\") pod \"kube-auth-proxy-5b78d9f867-wc99x\" (UID: \"0bcc10ec-7c22-44dc-b85a-00b4e3adf48e\") " pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"
Jul 23 21:57:59.672792 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.672793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4xn8\" (UniqueName: \"kubernetes.io/projected/0bcc10ec-7c22-44dc-b85a-00b4e3adf48e-kube-api-access-z4xn8\") pod \"kube-auth-proxy-5b78d9f867-wc99x\" (UID: \"0bcc10ec-7c22-44dc-b85a-00b4e3adf48e\") " pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"
Jul 23 21:57:59.675208 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.675175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bcc10ec-7c22-44dc-b85a-00b4e3adf48e-tmp\") pod \"kube-auth-proxy-5b78d9f867-wc99x\" (UID: \"0bcc10ec-7c22-44dc-b85a-00b4e3adf48e\") " pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"
Jul 23 21:57:59.675433 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.675415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bcc10ec-7c22-44dc-b85a-00b4e3adf48e-tls-certs\") pod \"kube-auth-proxy-5b78d9f867-wc99x\" (UID: \"0bcc10ec-7c22-44dc-b85a-00b4e3adf48e\") " pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"
Jul 23 21:57:59.680940 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.680909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4xn8\" (UniqueName: \"kubernetes.io/projected/0bcc10ec-7c22-44dc-b85a-00b4e3adf48e-kube-api-access-z4xn8\") pod \"kube-auth-proxy-5b78d9f867-wc99x\" (UID: \"0bcc10ec-7c22-44dc-b85a-00b4e3adf48e\") " pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"
Jul 23 21:57:59.726809 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.726757    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:59.778051 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.778009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"
Jul 23 21:57:59.827158 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.827075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:57:59.923842 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:57:59.923753    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x"]
Jul 23 21:57:59.926919 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:57:59.926884    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcc10ec_7c22_44dc_b85a_00b4e3adf48e.slice/crio-980f7063114cad899ec1da3b7564d9e4911f577d25569320673fab1560d2e02e WatchSource:0}: Error finding container 980f7063114cad899ec1da3b7564d9e4911f577d25569320673fab1560d2e02e: Status 404 returned error can't find the container with id 980f7063114cad899ec1da3b7564d9e4911f577d25569320673fab1560d2e02e
Jul 23 21:58:00.177626 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:00.177540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:00.560531 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:00.560479    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x" event={"ID":"0bcc10ec-7c22-44dc-b85a-00b4e3adf48e","Type":"ContainerStarted","Data":"980f7063114cad899ec1da3b7564d9e4911f577d25569320673fab1560d2e02e"}
Jul 23 21:58:00.908466 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:00.908334    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:00.999643 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:00.999597    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:02.072975 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:02.072939    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:02.569597 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:02.569561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x" event={"ID":"0bcc10ec-7c22-44dc-b85a-00b4e3adf48e","Type":"ContainerStarted","Data":"e7207057cb3bc084b41372d01f793bbb1cad1c46208dea6c3bf56e520ac89732"}
Jul 23 21:58:02.587813 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:02.587753    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b78d9f867-wc99x" podStartSLOduration=1.492143201 podStartE2EDuration="3.587735495s" totalImagesPullingTime="2.095592294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:58:02.587006704 +0000 UTC m=+377.977399320" watchObservedRunningTime="2026-07-23 21:58:02.587735495 +0000 UTC m=+377.978128112"
Jul 23 21:58:02.987615 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:02.987578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:03.046353 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:03.046318    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:05.038258 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:05.038218    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:05.155511 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:05.155476    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:06.202303 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:06.202267    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:06.327241 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:06.327182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:07.560536 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:07.560498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:07.838614 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:07.838526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:07.959519 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:07.959486    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:10.918543 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:10.918505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:11.036736 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:11.036698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:11.332335 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:11.332306    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:11.534662 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:11.534623    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:12.271580 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:12.271541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:12.370740 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:12.370695    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:12.473675 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:12.473638    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:14.685546 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:14.685508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:55.046395 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:55.046338    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:55.148353 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:55.148319    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:58:55.266739 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:58:55.266702    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:17.651095 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:17.651001    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:17.751462 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:17.751425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:18.333763 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:18.333728    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:21.110075 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:21.110035    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:24.515799 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:24.515761    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:25.330239 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:25.330206    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:26.755370 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:26.755330    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:29.305390 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:29.305356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:30.825584 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:30.825541    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:32.312003 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:32.311968    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:32.410552 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:32.410506    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:32.689530 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:32.689432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:32.752201 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:32.752167    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:36.095400 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:36.095357    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:36.155926 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:36.155886    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:36.615925 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:36.615892    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:36.928593 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:36.928485    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:37.410784 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:37.410747    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:40.093091 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:40.093054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:40.151744 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:40.151711    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:46.840441 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:46.840393    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:46.946766 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:46.946736    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:47.105805 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.105730    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"]
Jul 23 21:59:47.108144 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.108116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 21:59:47.112291 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.112262    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 21:59:47.112441 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.112272    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 21:59:47.112441 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.112273    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hkd2b\""
Jul 23 21:59:47.135409 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.135379    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"]
Jul 23 21:59:47.233092 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.233047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a3afcf-fb02-460f-bd32-2fa85fb56572-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dlvmv\" (UID: \"94a3afcf-fb02-460f-bd32-2fa85fb56572\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 21:59:47.233279 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.233166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2c72\" (UniqueName: \"kubernetes.io/projected/94a3afcf-fb02-460f-bd32-2fa85fb56572-kube-api-access-w2c72\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dlvmv\" (UID: \"94a3afcf-fb02-460f-bd32-2fa85fb56572\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 21:59:47.334242 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.334204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a3afcf-fb02-460f-bd32-2fa85fb56572-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dlvmv\" (UID: \"94a3afcf-fb02-460f-bd32-2fa85fb56572\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 21:59:47.334415 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.334260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2c72\" (UniqueName: \"kubernetes.io/projected/94a3afcf-fb02-460f-bd32-2fa85fb56572-kube-api-access-w2c72\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dlvmv\" (UID: \"94a3afcf-fb02-460f-bd32-2fa85fb56572\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 21:59:47.334607 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.334586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a3afcf-fb02-460f-bd32-2fa85fb56572-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dlvmv\" (UID: \"94a3afcf-fb02-460f-bd32-2fa85fb56572\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 21:59:47.345885 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.345842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2c72\" (UniqueName: \"kubernetes.io/projected/94a3afcf-fb02-460f-bd32-2fa85fb56572-kube-api-access-w2c72\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dlvmv\" (UID: \"94a3afcf-fb02-460f-bd32-2fa85fb56572\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 21:59:47.420149 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.420054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 21:59:47.567967 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.567937    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"]
Jul 23 21:59:47.571539 ip-10-0-135-91 kubenswrapper[2579]: W0723 21:59:47.571509    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a3afcf_fb02_460f_bd32_2fa85fb56572.slice/crio-2c5a4d1f82f1039e7504ad3b1f09f10a01db3b691087e9f3ae1314fb6aee1bae WatchSource:0}: Error finding container 2c5a4d1f82f1039e7504ad3b1f09f10a01db3b691087e9f3ae1314fb6aee1bae: Status 404 returned error can't find the container with id 2c5a4d1f82f1039e7504ad3b1f09f10a01db3b691087e9f3ae1314fb6aee1bae
Jul 23 21:59:47.929599 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:47.929564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv" event={"ID":"94a3afcf-fb02-460f-bd32-2fa85fb56572","Type":"ContainerStarted","Data":"2c5a4d1f82f1039e7504ad3b1f09f10a01db3b691087e9f3ae1314fb6aee1bae"}
Jul 23 21:59:48.487404 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:48.487367    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:48.584607 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:48.584576    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:48.684297 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:48.684263    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:48.784680 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:48.784587    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:50.624523 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:50.624481    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:50.727598 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:50.727550    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:51.229714 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:51.229639    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:51.326397 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:51.326355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:52.952498 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:52.952467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv" event={"ID":"94a3afcf-fb02-460f-bd32-2fa85fb56572","Type":"ContainerStarted","Data":"27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa"}
Jul 23 21:59:52.952913 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:52.952593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 21:59:52.973193 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:52.973145    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv" podStartSLOduration=1.459754598 podStartE2EDuration="5.97312934s" totalImagesPullingTime="4.513374742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 21:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 21:59:52.972878013 +0000 UTC m=+488.363270621" watchObservedRunningTime="2026-07-23 21:59:52.97312934 +0000 UTC m=+488.363521956"
Jul 23 21:59:53.628610 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:53.628561    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:53.730619 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:53.730582    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:58.877032 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:58.876989    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 21:59:58.964595 ip-10-0-135-91 kubenswrapper[2579]: I0723 21:59:58.964551    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:03.959181 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:03.959140    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 22:00:05.452328 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:05.452292    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:05.554385 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:05.554348    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:07.459329 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:07.459294    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:16.429856 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:16.429816    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"]
Jul 23 22:00:16.430290 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:16.430042    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv" podUID="94a3afcf-fb02-460f-bd32-2fa85fb56572" containerName="manager" containerID="cri-o://27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa" gracePeriod=10
Jul 23 22:00:16.672433 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:16.672408    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 22:00:16.744946 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:16.744911    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a3afcf-fb02-460f-bd32-2fa85fb56572-extensions-socket-volume\") pod \"94a3afcf-fb02-460f-bd32-2fa85fb56572\" (UID: \"94a3afcf-fb02-460f-bd32-2fa85fb56572\") "
Jul 23 22:00:16.744946 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:16.744955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2c72\" (UniqueName: \"kubernetes.io/projected/94a3afcf-fb02-460f-bd32-2fa85fb56572-kube-api-access-w2c72\") pod \"94a3afcf-fb02-460f-bd32-2fa85fb56572\" (UID: \"94a3afcf-fb02-460f-bd32-2fa85fb56572\") "
Jul 23 22:00:16.745390 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:16.745357    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94a3afcf-fb02-460f-bd32-2fa85fb56572-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "94a3afcf-fb02-460f-bd32-2fa85fb56572" (UID: "94a3afcf-fb02-460f-bd32-2fa85fb56572"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:00:16.747288 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:16.747262    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a3afcf-fb02-460f-bd32-2fa85fb56572-kube-api-access-w2c72" (OuterVolumeSpecName: "kube-api-access-w2c72") pod "94a3afcf-fb02-460f-bd32-2fa85fb56572" (UID: "94a3afcf-fb02-460f-bd32-2fa85fb56572"). InnerVolumeSpecName "kube-api-access-w2c72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:00:16.846481 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:16.846440    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94a3afcf-fb02-460f-bd32-2fa85fb56572-extensions-socket-volume\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:00:16.846481 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:16.846473    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2c72\" (UniqueName: \"kubernetes.io/projected/94a3afcf-fb02-460f-bd32-2fa85fb56572-kube-api-access-w2c72\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:00:17.045354 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:17.045262    2579 generic.go:356] "Generic (PLEG): container finished" podID="94a3afcf-fb02-460f-bd32-2fa85fb56572" containerID="27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa" exitCode=0
Jul 23 22:00:17.045354 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:17.045313    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"
Jul 23 22:00:17.045354 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:17.045337    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv" event={"ID":"94a3afcf-fb02-460f-bd32-2fa85fb56572","Type":"ContainerDied","Data":"27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa"}
Jul 23 22:00:17.045607 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:17.045391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv" event={"ID":"94a3afcf-fb02-460f-bd32-2fa85fb56572","Type":"ContainerDied","Data":"2c5a4d1f82f1039e7504ad3b1f09f10a01db3b691087e9f3ae1314fb6aee1bae"}
Jul 23 22:00:17.045607 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:17.045420    2579 scope.go:117] "RemoveContainer" containerID="27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa"
Jul 23 22:00:17.054557 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:17.054533    2579 scope.go:117] "RemoveContainer" containerID="27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa"
Jul 23 22:00:17.054847 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:00:17.054826    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa\": container with ID starting with 27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa not found: ID does not exist" containerID="27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa"
Jul 23 22:00:17.054938 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:17.054917    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa"} err="failed to get container status \"27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa\": rpc error: code = NotFound desc = could not find container \"27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa\": container with ID starting with 27fbdb7bbfe767959db3ce8a8cc14c771061109127c04822aba7a5f68ef047aa not found: ID does not exist"
Jul 23 22:00:17.071957 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:17.071907    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"]
Jul 23 22:00:17.077279 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:17.077247    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dlvmv"]
Jul 23 22:00:17.110704 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:17.110672    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94a3afcf-fb02-460f-bd32-2fa85fb56572" path="/var/lib/kubelet/pods/94a3afcf-fb02-460f-bd32-2fa85fb56572/volumes"
Jul 23 22:00:20.410595 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:20.410559    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:20.854696 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:20.854655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:34.197420 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.197388    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"]
Jul 23 22:00:34.197906 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.197666    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94a3afcf-fb02-460f-bd32-2fa85fb56572" containerName="manager"
Jul 23 22:00:34.197906 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.197678    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="94a3afcf-fb02-460f-bd32-2fa85fb56572" containerName="manager"
Jul 23 22:00:34.197906 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.197738    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="94a3afcf-fb02-460f-bd32-2fa85fb56572" containerName="manager"
Jul 23 22:00:34.200121 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.200095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:34.200496 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.200417    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:34.202535 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.202508    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 22:00:34.202619 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.202514    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 22:00:34.203184 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.203162    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zk25n\""
Jul 23 22:00:34.203255 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.203163    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 22:00:34.209494 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.209470    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"]
Jul 23 22:00:34.282723 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.282659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j5vr\" (UniqueName: \"kubernetes.io/projected/8c4862ae-53a6-470f-9d02-578c40154cc8-kube-api-access-4j5vr\") pod \"limitador-limitador-64c94fd64f-ph7w2\" (UID: \"8c4862ae-53a6-470f-9d02-578c40154cc8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:34.282926 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.282791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c4862ae-53a6-470f-9d02-578c40154cc8-config-file\") pod \"limitador-limitador-64c94fd64f-ph7w2\" (UID: \"8c4862ae-53a6-470f-9d02-578c40154cc8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:34.295908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.295842    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"]
Jul 23 22:00:34.304335 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.304298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:34.383202 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.383166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c4862ae-53a6-470f-9d02-578c40154cc8-config-file\") pod \"limitador-limitador-64c94fd64f-ph7w2\" (UID: \"8c4862ae-53a6-470f-9d02-578c40154cc8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:34.383414 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.383246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j5vr\" (UniqueName: \"kubernetes.io/projected/8c4862ae-53a6-470f-9d02-578c40154cc8-kube-api-access-4j5vr\") pod \"limitador-limitador-64c94fd64f-ph7w2\" (UID: \"8c4862ae-53a6-470f-9d02-578c40154cc8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:34.383414 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.383327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:34.383940 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.383917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c4862ae-53a6-470f-9d02-578c40154cc8-config-file\") pod \"limitador-limitador-64c94fd64f-ph7w2\" (UID: \"8c4862ae-53a6-470f-9d02-578c40154cc8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:34.392365 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.392340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j5vr\" (UniqueName: \"kubernetes.io/projected/8c4862ae-53a6-470f-9d02-578c40154cc8-kube-api-access-4j5vr\") pod \"limitador-limitador-64c94fd64f-ph7w2\" (UID: \"8c4862ae-53a6-470f-9d02-578c40154cc8\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:34.510503 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.510457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:34.646658 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:34.646603    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"]
Jul 23 22:00:34.649496 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:00:34.649463    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4862ae_53a6_470f_9d02_578c40154cc8.slice/crio-eddf741ebc2e4fbb88e7eaa81ced30806a9ae925556443cc689b62e757700175 WatchSource:0}: Error finding container eddf741ebc2e4fbb88e7eaa81ced30806a9ae925556443cc689b62e757700175: Status 404 returned error can't find the container with id eddf741ebc2e4fbb88e7eaa81ced30806a9ae925556443cc689b62e757700175
Jul 23 22:00:35.026671 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.026640    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:35.092446 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.092406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:35.112016 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.111981    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-pbpzd"]
Jul 23 22:00:35.115291 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.115273    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pbpzd"
Jul 23 22:00:35.117351 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.117327    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zr4mt\""
Jul 23 22:00:35.120748 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.120724    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pbpzd"]
Jul 23 22:00:35.125546 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.125517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2" event={"ID":"8c4862ae-53a6-470f-9d02-578c40154cc8","Type":"ContainerStarted","Data":"eddf741ebc2e4fbb88e7eaa81ced30806a9ae925556443cc689b62e757700175"}
Jul 23 22:00:35.191406 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.191320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th5gq\" (UniqueName: \"kubernetes.io/projected/8976f486-d1b8-47a7-a109-e20623ba7a9f-kube-api-access-th5gq\") pod \"authorino-75d556bd77-pbpzd\" (UID: \"8976f486-d1b8-47a7-a109-e20623ba7a9f\") " pod="kuadrant-system/authorino-75d556bd77-pbpzd"
Jul 23 22:00:35.292338 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.292241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th5gq\" (UniqueName: \"kubernetes.io/projected/8976f486-d1b8-47a7-a109-e20623ba7a9f-kube-api-access-th5gq\") pod \"authorino-75d556bd77-pbpzd\" (UID: \"8976f486-d1b8-47a7-a109-e20623ba7a9f\") " pod="kuadrant-system/authorino-75d556bd77-pbpzd"
Jul 23 22:00:35.299879 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.299835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th5gq\" (UniqueName: \"kubernetes.io/projected/8976f486-d1b8-47a7-a109-e20623ba7a9f-kube-api-access-th5gq\") pod \"authorino-75d556bd77-pbpzd\" (UID: \"8976f486-d1b8-47a7-a109-e20623ba7a9f\") " pod="kuadrant-system/authorino-75d556bd77-pbpzd"
Jul 23 22:00:35.426102 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.426065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pbpzd"
Jul 23 22:00:35.627342 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:35.626747    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pbpzd"]
Jul 23 22:00:35.631994 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:00:35.631950    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8976f486_d1b8_47a7_a109_e20623ba7a9f.slice/crio-37e54569cfb3b6343441e57d3e5b992d6adbea9905e55eebc8a802825d5ff664 WatchSource:0}: Error finding container 37e54569cfb3b6343441e57d3e5b992d6adbea9905e55eebc8a802825d5ff664: Status 404 returned error can't find the container with id 37e54569cfb3b6343441e57d3e5b992d6adbea9905e55eebc8a802825d5ff664
Jul 23 22:00:36.136676 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:36.136626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pbpzd" event={"ID":"8976f486-d1b8-47a7-a109-e20623ba7a9f","Type":"ContainerStarted","Data":"37e54569cfb3b6343441e57d3e5b992d6adbea9905e55eebc8a802825d5ff664"}
Jul 23 22:00:36.557597 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:36.557543    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:38.150027 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:38.149984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2" event={"ID":"8c4862ae-53a6-470f-9d02-578c40154cc8","Type":"ContainerStarted","Data":"5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb"}
Jul 23 22:00:38.150501 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:38.150119    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:38.168018 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:38.167966    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2" podStartSLOduration=1.30747171 podStartE2EDuration="4.167947881s" totalImagesPullingTime="2.860476171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:00:38.167391736 +0000 UTC m=+533.557784353" watchObservedRunningTime="2026-07-23 22:00:38.167947881 +0000 UTC m=+533.558340501"
Jul 23 22:00:40.159558 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:40.159517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pbpzd" event={"ID":"8976f486-d1b8-47a7-a109-e20623ba7a9f","Type":"ContainerStarted","Data":"9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69"}
Jul 23 22:00:40.176274 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:40.176222    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-pbpzd" podStartSLOduration=1.238912507 podStartE2EDuration="5.176201014s" totalImagesPullingTime="3.937288507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:00:40.175284838 +0000 UTC m=+535.565677455" watchObservedRunningTime="2026-07-23 22:00:40.176201014 +0000 UTC m=+535.566593630"
Jul 23 22:00:49.155705 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:49.155674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:50.625913 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:50.625872    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"]
Jul 23 22:00:50.626339 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:50.626083    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2" podUID="8c4862ae-53a6-470f-9d02-578c40154cc8" containerName="limitador" containerID="cri-o://5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb" gracePeriod=30
Jul 23 22:00:51.170160 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.170134    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:51.198822 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.198746    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c4862ae-53a6-470f-9d02-578c40154cc8" containerID="5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb" exitCode=0
Jul 23 22:00:51.198822 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.198807    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"
Jul 23 22:00:51.198997 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.198830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2" event={"ID":"8c4862ae-53a6-470f-9d02-578c40154cc8","Type":"ContainerDied","Data":"5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb"}
Jul 23 22:00:51.198997 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.198904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7w2" event={"ID":"8c4862ae-53a6-470f-9d02-578c40154cc8","Type":"ContainerDied","Data":"eddf741ebc2e4fbb88e7eaa81ced30806a9ae925556443cc689b62e757700175"}
Jul 23 22:00:51.198997 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.198928    2579 scope.go:117] "RemoveContainer" containerID="5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb"
Jul 23 22:00:51.207054 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.207037    2579 scope.go:117] "RemoveContainer" containerID="5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb"
Jul 23 22:00:51.207307 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:00:51.207290    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb\": container with ID starting with 5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb not found: ID does not exist" containerID="5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb"
Jul 23 22:00:51.207386 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.207316    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb"} err="failed to get container status \"5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb\": rpc error: code = NotFound desc = could not find container \"5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb\": container with ID starting with 5493c0290bc0d84856e6e9319596a9c80fb45e47ed8c3b403f3e6e4a8e1f1fcb not found: ID does not exist"
Jul 23 22:00:51.336920 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.336853    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4j5vr\" (UniqueName: \"kubernetes.io/projected/8c4862ae-53a6-470f-9d02-578c40154cc8-kube-api-access-4j5vr\") pod \"8c4862ae-53a6-470f-9d02-578c40154cc8\" (UID: \"8c4862ae-53a6-470f-9d02-578c40154cc8\") "
Jul 23 22:00:51.337100 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.336974    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c4862ae-53a6-470f-9d02-578c40154cc8-config-file\") pod \"8c4862ae-53a6-470f-9d02-578c40154cc8\" (UID: \"8c4862ae-53a6-470f-9d02-578c40154cc8\") "
Jul 23 22:00:51.337318 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.337296    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4862ae-53a6-470f-9d02-578c40154cc8-config-file" (OuterVolumeSpecName: "config-file") pod "8c4862ae-53a6-470f-9d02-578c40154cc8" (UID: "8c4862ae-53a6-470f-9d02-578c40154cc8"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:00:51.339146 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.339125    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4862ae-53a6-470f-9d02-578c40154cc8-kube-api-access-4j5vr" (OuterVolumeSpecName: "kube-api-access-4j5vr") pod "8c4862ae-53a6-470f-9d02-578c40154cc8" (UID: "8c4862ae-53a6-470f-9d02-578c40154cc8"). InnerVolumeSpecName "kube-api-access-4j5vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:00:51.438068 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.438037    2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c4862ae-53a6-470f-9d02-578c40154cc8-config-file\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:00:51.438068 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.438061    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j5vr\" (UniqueName: \"kubernetes.io/projected/8c4862ae-53a6-470f-9d02-578c40154cc8-kube-api-access-4j5vr\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:00:51.520755 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.520716    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"]
Jul 23 22:00:51.524643 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:51.524604    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7w2"]
Jul 23 22:00:53.111233 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:53.111198    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c4862ae-53a6-470f-9d02-578c40154cc8" path="/var/lib/kubelet/pods/8c4862ae-53a6-470f-9d02-578c40154cc8/volumes"
Jul 23 22:00:54.095581 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.095534    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:54.190505 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.190472    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:54.297390 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.297354    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:54.548482 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.548420    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"]
Jul 23 22:00:54.548737 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.548723    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c4862ae-53a6-470f-9d02-578c40154cc8" containerName="limitador"
Jul 23 22:00:54.548775 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.548739    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c4862ae-53a6-470f-9d02-578c40154cc8" containerName="limitador"
Jul 23 22:00:54.548809 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.548800    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c4862ae-53a6-470f-9d02-578c40154cc8" containerName="limitador"
Jul 23 22:00:54.551783 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.551766    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"
Jul 23 22:00:54.554230 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.554192    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 23 22:00:54.554230 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.554212    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-65rwc\""
Jul 23 22:00:54.554410 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.554251    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 22:00:54.555033 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.555009    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 22:00:54.562257 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.562232    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"]
Jul 23 22:00:54.582147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.582111    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:00:54.665517 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.665477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226b7\" (UniqueName: \"kubernetes.io/projected/c1bdfa83-6de1-43fa-9432-8da3566bbc73-kube-api-access-226b7\") pod \"postgres-69fb4594cf-fhnpl\" (UID: \"c1bdfa83-6de1-43fa-9432-8da3566bbc73\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"
Jul 23 22:00:54.665517 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.665511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c1bdfa83-6de1-43fa-9432-8da3566bbc73-data\") pod \"postgres-69fb4594cf-fhnpl\" (UID: \"c1bdfa83-6de1-43fa-9432-8da3566bbc73\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"
Jul 23 22:00:54.766797 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.766758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226b7\" (UniqueName: \"kubernetes.io/projected/c1bdfa83-6de1-43fa-9432-8da3566bbc73-kube-api-access-226b7\") pod \"postgres-69fb4594cf-fhnpl\" (UID: \"c1bdfa83-6de1-43fa-9432-8da3566bbc73\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"
Jul 23 22:00:54.766797 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.766797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c1bdfa83-6de1-43fa-9432-8da3566bbc73-data\") pod \"postgres-69fb4594cf-fhnpl\" (UID: \"c1bdfa83-6de1-43fa-9432-8da3566bbc73\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"
Jul 23 22:00:54.767203 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.767185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c1bdfa83-6de1-43fa-9432-8da3566bbc73-data\") pod \"postgres-69fb4594cf-fhnpl\" (UID: \"c1bdfa83-6de1-43fa-9432-8da3566bbc73\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"
Jul 23 22:00:54.775020 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.775003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-226b7\" (UniqueName: \"kubernetes.io/projected/c1bdfa83-6de1-43fa-9432-8da3566bbc73-kube-api-access-226b7\") pod \"postgres-69fb4594cf-fhnpl\" (UID: \"c1bdfa83-6de1-43fa-9432-8da3566bbc73\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"
Jul 23 22:00:54.862784 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.862698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"
Jul 23 22:00:54.996228 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:54.996201    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"]
Jul 23 22:00:54.998822 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:00:54.998782    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bdfa83_6de1_43fa_9432_8da3566bbc73.slice/crio-be0e2ee733a5015c8a7459bcce7197f9108d2be85c47e48eb388ba7fa0c8ef2f WatchSource:0}: Error finding container be0e2ee733a5015c8a7459bcce7197f9108d2be85c47e48eb388ba7fa0c8ef2f: Status 404 returned error can't find the container with id be0e2ee733a5015c8a7459bcce7197f9108d2be85c47e48eb388ba7fa0c8ef2f
Jul 23 22:00:55.215409 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:00:55.215371    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl" event={"ID":"c1bdfa83-6de1-43fa-9432-8da3566bbc73","Type":"ContainerStarted","Data":"be0e2ee733a5015c8a7459bcce7197f9108d2be85c47e48eb388ba7fa0c8ef2f"}
Jul 23 22:01:00.240985 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:00.240936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl" event={"ID":"c1bdfa83-6de1-43fa-9432-8da3566bbc73","Type":"ContainerStarted","Data":"4f5dcede9635b52850d73da06c59bc2550fdf22eaa11b68aef176561b502815f"}
Jul 23 22:01:00.241391 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:00.241040    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"
Jul 23 22:01:00.258622 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:00.258570    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl" podStartSLOduration=1.339260009 podStartE2EDuration="6.258554167s" totalImagesPullingTime="4.919294158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:01:00.257256404 +0000 UTC m=+555.647649084" watchObservedRunningTime="2026-07-23 22:01:00.258554167 +0000 UTC m=+555.648946782"
Jul 23 22:01:06.274940 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:06.274905    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-fhnpl"
Jul 23 22:01:06.612999 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:06.612916    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:06.775572 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:06.775540    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:06.841542 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:06.841510    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-m7pht"]
Jul 23 22:01:06.843699 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:06.843683    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-m7pht"
Jul 23 22:01:06.851884 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:06.851839    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-m7pht"]
Jul 23 22:01:06.866369 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:06.866288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njpbw\" (UniqueName: \"kubernetes.io/projected/0f940675-50be-4361-afb0-ad62e15dfe5e-kube-api-access-njpbw\") pod \"authorino-5d4b67c446-m7pht\" (UID: \"0f940675-50be-4361-afb0-ad62e15dfe5e\") " pod="kuadrant-system/authorino-5d4b67c446-m7pht"
Jul 23 22:01:06.967044 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:06.967002    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njpbw\" (UniqueName: \"kubernetes.io/projected/0f940675-50be-4361-afb0-ad62e15dfe5e-kube-api-access-njpbw\") pod \"authorino-5d4b67c446-m7pht\" (UID: \"0f940675-50be-4361-afb0-ad62e15dfe5e\") " pod="kuadrant-system/authorino-5d4b67c446-m7pht"
Jul 23 22:01:06.975684 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:06.975644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njpbw\" (UniqueName: \"kubernetes.io/projected/0f940675-50be-4361-afb0-ad62e15dfe5e-kube-api-access-njpbw\") pod \"authorino-5d4b67c446-m7pht\" (UID: \"0f940675-50be-4361-afb0-ad62e15dfe5e\") " pod="kuadrant-system/authorino-5d4b67c446-m7pht"
Jul 23 22:01:07.080013 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.079971    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-m7pht"]
Jul 23 22:01:07.080270 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.080252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-m7pht"
Jul 23 22:01:07.103608 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.103229    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d8f9548d9-jhhjw"]
Jul 23 22:01:07.125500 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.125397    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d8f9548d9-jhhjw"]
Jul 23 22:01:07.125675 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.125518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d8f9548d9-jhhjw"
Jul 23 22:01:07.169669 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.169635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqpp4\" (UniqueName: \"kubernetes.io/projected/b0474400-dab6-4a72-b0af-f072340c18b7-kube-api-access-fqpp4\") pod \"authorino-6d8f9548d9-jhhjw\" (UID: \"b0474400-dab6-4a72-b0af-f072340c18b7\") " pod="kuadrant-system/authorino-6d8f9548d9-jhhjw"
Jul 23 22:01:07.201438 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.201402    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d8f9548d9-jhhjw"]
Jul 23 22:01:07.201718 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:01:07.201694    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fqpp4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d8f9548d9-jhhjw" podUID="b0474400-dab6-4a72-b0af-f072340c18b7"
Jul 23 22:01:07.229659 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.229624    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68ffc9f8df-g57cd"]
Jul 23 22:01:07.243020 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:01:07.242991    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f940675_50be_4361_afb0_ad62e15dfe5e.slice/crio-06d854c61cb40da177306854f45cc32784a0a476cec9fb0a675b8cb22c5a7ea1 WatchSource:0}: Error finding container 06d854c61cb40da177306854f45cc32784a0a476cec9fb0a675b8cb22c5a7ea1: Status 404 returned error can't find the container with id 06d854c61cb40da177306854f45cc32784a0a476cec9fb0a675b8cb22c5a7ea1
Jul 23 22:01:07.248148 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.248076    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-m7pht"]
Jul 23 22:01:07.248279 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.248171    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68ffc9f8df-g57cd"]
Jul 23 22:01:07.248279 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.248224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68ffc9f8df-g57cd"
Jul 23 22:01:07.250746 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.250722    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 23 22:01:07.266653 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.266632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d8f9548d9-jhhjw"
Jul 23 22:01:07.266653 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.266638    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-m7pht" event={"ID":"0f940675-50be-4361-afb0-ad62e15dfe5e","Type":"ContainerStarted","Data":"06d854c61cb40da177306854f45cc32784a0a476cec9fb0a675b8cb22c5a7ea1"}
Jul 23 22:01:07.270183 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.270164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-tls-cert\") pod \"authorino-68ffc9f8df-g57cd\" (UID: \"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2\") " pod="kuadrant-system/authorino-68ffc9f8df-g57cd"
Jul 23 22:01:07.270314 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.270215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgtxf\" (UniqueName: \"kubernetes.io/projected/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-kube-api-access-vgtxf\") pod \"authorino-68ffc9f8df-g57cd\" (UID: \"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2\") " pod="kuadrant-system/authorino-68ffc9f8df-g57cd"
Jul 23 22:01:07.270314 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.270298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqpp4\" (UniqueName: \"kubernetes.io/projected/b0474400-dab6-4a72-b0af-f072340c18b7-kube-api-access-fqpp4\") pod \"authorino-6d8f9548d9-jhhjw\" (UID: \"b0474400-dab6-4a72-b0af-f072340c18b7\") " pod="kuadrant-system/authorino-6d8f9548d9-jhhjw"
Jul 23 22:01:07.278925 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.278886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqpp4\" (UniqueName: \"kubernetes.io/projected/b0474400-dab6-4a72-b0af-f072340c18b7-kube-api-access-fqpp4\") pod \"authorino-6d8f9548d9-jhhjw\" (UID: \"b0474400-dab6-4a72-b0af-f072340c18b7\") " pod="kuadrant-system/authorino-6d8f9548d9-jhhjw"
Jul 23 22:01:07.280437 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.280417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d8f9548d9-jhhjw"
Jul 23 22:01:07.370964 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.370924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqpp4\" (UniqueName: \"kubernetes.io/projected/b0474400-dab6-4a72-b0af-f072340c18b7-kube-api-access-fqpp4\") pod \"b0474400-dab6-4a72-b0af-f072340c18b7\" (UID: \"b0474400-dab6-4a72-b0af-f072340c18b7\") "
Jul 23 22:01:07.371131 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.371020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-tls-cert\") pod \"authorino-68ffc9f8df-g57cd\" (UID: \"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2\") " pod="kuadrant-system/authorino-68ffc9f8df-g57cd"
Jul 23 22:01:07.371131 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.371052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgtxf\" (UniqueName: \"kubernetes.io/projected/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-kube-api-access-vgtxf\") pod \"authorino-68ffc9f8df-g57cd\" (UID: \"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2\") " pod="kuadrant-system/authorino-68ffc9f8df-g57cd"
Jul 23 22:01:07.373246 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.373216    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0474400-dab6-4a72-b0af-f072340c18b7-kube-api-access-fqpp4" (OuterVolumeSpecName: "kube-api-access-fqpp4") pod "b0474400-dab6-4a72-b0af-f072340c18b7" (UID: "b0474400-dab6-4a72-b0af-f072340c18b7"). InnerVolumeSpecName "kube-api-access-fqpp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:01:07.373559 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.373539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-tls-cert\") pod \"authorino-68ffc9f8df-g57cd\" (UID: \"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2\") " pod="kuadrant-system/authorino-68ffc9f8df-g57cd"
Jul 23 22:01:07.378595 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.378539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgtxf\" (UniqueName: \"kubernetes.io/projected/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-kube-api-access-vgtxf\") pod \"authorino-68ffc9f8df-g57cd\" (UID: \"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2\") " pod="kuadrant-system/authorino-68ffc9f8df-g57cd"
Jul 23 22:01:07.471553 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.471524    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqpp4\" (UniqueName: \"kubernetes.io/projected/b0474400-dab6-4a72-b0af-f072340c18b7-kube-api-access-fqpp4\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:01:07.558451 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.558411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68ffc9f8df-g57cd"
Jul 23 22:01:07.710882 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:07.710833    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68ffc9f8df-g57cd"]
Jul 23 22:01:07.714239 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:01:07.714212    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97d5577f_a446_4cbc_91f0_8fbf7a3cb4a2.slice/crio-fd305c58e732928d60525a5c255f34e2d8417d5d8f79ffa86d4f0ac4bc873317 WatchSource:0}: Error finding container fd305c58e732928d60525a5c255f34e2d8417d5d8f79ffa86d4f0ac4bc873317: Status 404 returned error can't find the container with id fd305c58e732928d60525a5c255f34e2d8417d5d8f79ffa86d4f0ac4bc873317
Jul 23 22:01:08.272740 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.272700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68ffc9f8df-g57cd" event={"ID":"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2","Type":"ContainerStarted","Data":"7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111"}
Jul 23 22:01:08.272914 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.272757    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68ffc9f8df-g57cd" event={"ID":"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2","Type":"ContainerStarted","Data":"fd305c58e732928d60525a5c255f34e2d8417d5d8f79ffa86d4f0ac4bc873317"}
Jul 23 22:01:08.274451 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.274421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d8f9548d9-jhhjw"
Jul 23 22:01:08.274572 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.274465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-m7pht" event={"ID":"0f940675-50be-4361-afb0-ad62e15dfe5e","Type":"ContainerStarted","Data":"3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db"}
Jul 23 22:01:08.274572 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.274480    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-m7pht" podUID="0f940675-50be-4361-afb0-ad62e15dfe5e" containerName="authorino" containerID="cri-o://3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db" gracePeriod=30
Jul 23 22:01:08.307821 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.307767    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-m7pht" podStartSLOduration=1.730663753 podStartE2EDuration="2.307748912s" totalImagesPullingTime="577.085159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:01:08.293133293 +0000 UTC m=+563.683525911" watchObservedRunningTime="2026-07-23 22:01:08.307748912 +0000 UTC m=+563.698141527"
Jul 23 22:01:08.315327 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.315268    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d8f9548d9-jhhjw"]
Jul 23 22:01:08.317623 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.317586    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d8f9548d9-jhhjw"]
Jul 23 22:01:08.554635 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.554599    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-m7pht"
Jul 23 22:01:08.580983 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.580948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njpbw\" (UniqueName: \"kubernetes.io/projected/0f940675-50be-4361-afb0-ad62e15dfe5e-kube-api-access-njpbw\") pod \"0f940675-50be-4361-afb0-ad62e15dfe5e\" (UID: \"0f940675-50be-4361-afb0-ad62e15dfe5e\") "
Jul 23 22:01:08.583063 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.583031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f940675-50be-4361-afb0-ad62e15dfe5e-kube-api-access-njpbw" (OuterVolumeSpecName: "kube-api-access-njpbw") pod "0f940675-50be-4361-afb0-ad62e15dfe5e" (UID: "0f940675-50be-4361-afb0-ad62e15dfe5e"). InnerVolumeSpecName "kube-api-access-njpbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:01:08.682493 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:08.682407    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njpbw\" (UniqueName: \"kubernetes.io/projected/0f940675-50be-4361-afb0-ad62e15dfe5e-kube-api-access-njpbw\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:01:09.111486 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.111454    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0474400-dab6-4a72-b0af-f072340c18b7" path="/var/lib/kubelet/pods/b0474400-dab6-4a72-b0af-f072340c18b7/volumes"
Jul 23 22:01:09.280210 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.280124    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f940675-50be-4361-afb0-ad62e15dfe5e" containerID="3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db" exitCode=0
Jul 23 22:01:09.280210 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.280166    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-m7pht"
Jul 23 22:01:09.280400 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.280222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-m7pht" event={"ID":"0f940675-50be-4361-afb0-ad62e15dfe5e","Type":"ContainerDied","Data":"3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db"}
Jul 23 22:01:09.280400 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.280261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-m7pht" event={"ID":"0f940675-50be-4361-afb0-ad62e15dfe5e","Type":"ContainerDied","Data":"06d854c61cb40da177306854f45cc32784a0a476cec9fb0a675b8cb22c5a7ea1"}
Jul 23 22:01:09.280400 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.280281    2579 scope.go:117] "RemoveContainer" containerID="3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db"
Jul 23 22:01:09.288335 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.288314    2579 scope.go:117] "RemoveContainer" containerID="3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db"
Jul 23 22:01:09.288624 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:01:09.288606    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db\": container with ID starting with 3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db not found: ID does not exist" containerID="3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db"
Jul 23 22:01:09.288699 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.288634    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db"} err="failed to get container status \"3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db\": rpc error: code = NotFound desc = could not find container \"3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db\": container with ID starting with 3e0429435edc9d09533b08eb1f46f53a7a1aa3827cd373e395c612866aa547db not found: ID does not exist"
Jul 23 22:01:09.295405 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.295382    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-m7pht"]
Jul 23 22:01:09.299712 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.299684    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-m7pht"]
Jul 23 22:01:09.315511 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.315453    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-68ffc9f8df-g57cd" podStartSLOduration=1.8729260669999999 podStartE2EDuration="2.315431981s" totalImagesPullingTime="442.505914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:01:09.31479733 +0000 UTC m=+564.705189937" watchObservedRunningTime="2026-07-23 22:01:09.315431981 +0000 UTC m=+564.705824599"
Jul 23 22:01:09.346327 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.346298    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pbpzd"]
Jul 23 22:01:09.346498 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.346477    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-pbpzd" podUID="8976f486-d1b8-47a7-a109-e20623ba7a9f" containerName="authorino" containerID="cri-o://9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69" gracePeriod=30
Jul 23 22:01:09.641392 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.641359    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pbpzd"
Jul 23 22:01:09.692574 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.692534    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-th5gq\" (UniqueName: \"kubernetes.io/projected/8976f486-d1b8-47a7-a109-e20623ba7a9f-kube-api-access-th5gq\") pod \"8976f486-d1b8-47a7-a109-e20623ba7a9f\" (UID: \"8976f486-d1b8-47a7-a109-e20623ba7a9f\") "
Jul 23 22:01:09.694745 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.694712    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8976f486-d1b8-47a7-a109-e20623ba7a9f-kube-api-access-th5gq" (OuterVolumeSpecName: "kube-api-access-th5gq") pod "8976f486-d1b8-47a7-a109-e20623ba7a9f" (UID: "8976f486-d1b8-47a7-a109-e20623ba7a9f"). InnerVolumeSpecName "kube-api-access-th5gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:01:09.794142 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:09.794097    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th5gq\" (UniqueName: \"kubernetes.io/projected/8976f486-d1b8-47a7-a109-e20623ba7a9f-kube-api-access-th5gq\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:01:10.286147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:10.286114    2579 generic.go:356] "Generic (PLEG): container finished" podID="8976f486-d1b8-47a7-a109-e20623ba7a9f" containerID="9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69" exitCode=0
Jul 23 22:01:10.286336 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:10.286163    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pbpzd"
Jul 23 22:01:10.286336 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:10.286223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pbpzd" event={"ID":"8976f486-d1b8-47a7-a109-e20623ba7a9f","Type":"ContainerDied","Data":"9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69"}
Jul 23 22:01:10.286336 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:10.286285    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pbpzd" event={"ID":"8976f486-d1b8-47a7-a109-e20623ba7a9f","Type":"ContainerDied","Data":"37e54569cfb3b6343441e57d3e5b992d6adbea9905e55eebc8a802825d5ff664"}
Jul 23 22:01:10.286336 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:10.286316    2579 scope.go:117] "RemoveContainer" containerID="9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69"
Jul 23 22:01:10.295083 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:10.294821    2579 scope.go:117] "RemoveContainer" containerID="9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69"
Jul 23 22:01:10.295218 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:01:10.295190    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69\": container with ID starting with 9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69 not found: ID does not exist" containerID="9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69"
Jul 23 22:01:10.295258 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:10.295236    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69"} err="failed to get container status \"9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69\": rpc error: code = NotFound desc = could not find container \"9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69\": container with ID starting with 9181a445eddb34fc832e199faa36115f0ce6bc4b67f172a1e6665dd0fb81ea69 not found: ID does not exist"
Jul 23 22:01:10.306608 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:10.306571    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pbpzd"]
Jul 23 22:01:10.308497 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:10.308471    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pbpzd"]
Jul 23 22:01:11.110805 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:11.110768    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f940675-50be-4361-afb0-ad62e15dfe5e" path="/var/lib/kubelet/pods/0f940675-50be-4361-afb0-ad62e15dfe5e/volumes"
Jul 23 22:01:11.111197 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:11.111171    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8976f486-d1b8-47a7-a109-e20623ba7a9f" path="/var/lib/kubelet/pods/8976f486-d1b8-47a7-a109-e20623ba7a9f/volumes"
Jul 23 22:01:14.081557 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:14.081524    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:14.198779 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:14.198742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:14.326412 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:14.326373    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:14.425447 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:14.425354    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:17.313445 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:17.313402    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:17.412219 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:17.412186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:45.006956 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:45.006931    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:01:45.009848 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:45.009826    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:01:49.567885 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:49.567830    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:49.704475 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:49.704442    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:50.165973 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.165935    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:50.250292 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.250257    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"]
Jul 23 22:01:50.250548 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.250536    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8976f486-d1b8-47a7-a109-e20623ba7a9f" containerName="authorino"
Jul 23 22:01:50.250599 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.250549    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8976f486-d1b8-47a7-a109-e20623ba7a9f" containerName="authorino"
Jul 23 22:01:50.250599 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.250566    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f940675-50be-4361-afb0-ad62e15dfe5e" containerName="authorino"
Jul 23 22:01:50.250599 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.250571    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f940675-50be-4361-afb0-ad62e15dfe5e" containerName="authorino"
Jul 23 22:01:50.250703 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.250620    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8976f486-d1b8-47a7-a109-e20623ba7a9f" containerName="authorino"
Jul 23 22:01:50.250703 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.250627    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f940675-50be-4361-afb0-ad62e15dfe5e" containerName="authorino"
Jul 23 22:01:50.253780 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.253756    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:50.256076 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.256051    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 23 22:01:50.256611 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.256594    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5vm4f\""
Jul 23 22:01:50.268509 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.268476    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"]
Jul 23 22:01:50.426672 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.426577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6twbw\" (UniqueName: \"kubernetes.io/projected/7886aac9-112c-4ad6-ae02-d7cff7963ce8-kube-api-access-6twbw\") pod \"maas-api-76bbd449ff-bf66q\" (UID: \"7886aac9-112c-4ad6-ae02-d7cff7963ce8\") " pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:50.426672 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.426627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7886aac9-112c-4ad6-ae02-d7cff7963ce8-openshift-service-ca\") pod \"maas-api-76bbd449ff-bf66q\" (UID: \"7886aac9-112c-4ad6-ae02-d7cff7963ce8\") " pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:50.426894 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.426721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7886aac9-112c-4ad6-ae02-d7cff7963ce8-maas-api-tls\") pod \"maas-api-76bbd449ff-bf66q\" (UID: \"7886aac9-112c-4ad6-ae02-d7cff7963ce8\") " pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:50.527358 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.527228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7886aac9-112c-4ad6-ae02-d7cff7963ce8-maas-api-tls\") pod \"maas-api-76bbd449ff-bf66q\" (UID: \"7886aac9-112c-4ad6-ae02-d7cff7963ce8\") " pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:50.527358 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.527299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6twbw\" (UniqueName: \"kubernetes.io/projected/7886aac9-112c-4ad6-ae02-d7cff7963ce8-kube-api-access-6twbw\") pod \"maas-api-76bbd449ff-bf66q\" (UID: \"7886aac9-112c-4ad6-ae02-d7cff7963ce8\") " pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:50.527358 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.527333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7886aac9-112c-4ad6-ae02-d7cff7963ce8-openshift-service-ca\") pod \"maas-api-76bbd449ff-bf66q\" (UID: \"7886aac9-112c-4ad6-ae02-d7cff7963ce8\") " pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:50.533934 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:01:50.527370    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 23 22:01:50.533934 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:01:50.527434    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7886aac9-112c-4ad6-ae02-d7cff7963ce8-maas-api-tls podName:7886aac9-112c-4ad6-ae02-d7cff7963ce8 nodeName:}" failed. No retries permitted until 2026-07-23 22:01:51.027417783 +0000 UTC m=+606.417810377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7886aac9-112c-4ad6-ae02-d7cff7963ce8-maas-api-tls") pod "maas-api-76bbd449ff-bf66q" (UID: "7886aac9-112c-4ad6-ae02-d7cff7963ce8") : secret "maas-api-serving-cert" not found
Jul 23 22:01:50.533934 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.527933    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7886aac9-112c-4ad6-ae02-d7cff7963ce8-openshift-service-ca\") pod \"maas-api-76bbd449ff-bf66q\" (UID: \"7886aac9-112c-4ad6-ae02-d7cff7963ce8\") " pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:50.541143 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.541112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6twbw\" (UniqueName: \"kubernetes.io/projected/7886aac9-112c-4ad6-ae02-d7cff7963ce8-kube-api-access-6twbw\") pod \"maas-api-76bbd449ff-bf66q\" (UID: \"7886aac9-112c-4ad6-ae02-d7cff7963ce8\") " pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:50.760264 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.760233    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:50.858250 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.858214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:01:50.889627 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.889588    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-6pshb"]
Jul 23 22:01:50.898024 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.897982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-6pshb"
Jul 23 22:01:50.900517 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.900492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gm464\""
Jul 23 22:01:50.900672 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.900524    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 23 22:01:50.903566 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:50.903536    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-6pshb"]
Jul 23 22:01:51.031546 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.031448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f667c912-b23d-4b18-be7b-6f4574133175-plugins-config-volume\") pod \"payload-processing-5bf47cd988-6pshb\" (UID: \"f667c912-b23d-4b18-be7b-6f4574133175\") " pod="openshift-ingress/payload-processing-5bf47cd988-6pshb"
Jul 23 22:01:51.031546 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.031512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njr8w\" (UniqueName: \"kubernetes.io/projected/f667c912-b23d-4b18-be7b-6f4574133175-kube-api-access-njr8w\") pod \"payload-processing-5bf47cd988-6pshb\" (UID: \"f667c912-b23d-4b18-be7b-6f4574133175\") " pod="openshift-ingress/payload-processing-5bf47cd988-6pshb"
Jul 23 22:01:51.031738 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.031593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7886aac9-112c-4ad6-ae02-d7cff7963ce8-maas-api-tls\") pod \"maas-api-76bbd449ff-bf66q\" (UID: \"7886aac9-112c-4ad6-ae02-d7cff7963ce8\") " pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:51.034120 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.034090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7886aac9-112c-4ad6-ae02-d7cff7963ce8-maas-api-tls\") pod \"maas-api-76bbd449ff-bf66q\" (UID: \"7886aac9-112c-4ad6-ae02-d7cff7963ce8\") " pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:51.133024 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.132980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njr8w\" (UniqueName: \"kubernetes.io/projected/f667c912-b23d-4b18-be7b-6f4574133175-kube-api-access-njr8w\") pod \"payload-processing-5bf47cd988-6pshb\" (UID: \"f667c912-b23d-4b18-be7b-6f4574133175\") " pod="openshift-ingress/payload-processing-5bf47cd988-6pshb"
Jul 23 22:01:51.133212 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.133073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f667c912-b23d-4b18-be7b-6f4574133175-plugins-config-volume\") pod \"payload-processing-5bf47cd988-6pshb\" (UID: \"f667c912-b23d-4b18-be7b-6f4574133175\") " pod="openshift-ingress/payload-processing-5bf47cd988-6pshb"
Jul 23 22:01:51.133763 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.133744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f667c912-b23d-4b18-be7b-6f4574133175-plugins-config-volume\") pod \"payload-processing-5bf47cd988-6pshb\" (UID: \"f667c912-b23d-4b18-be7b-6f4574133175\") " pod="openshift-ingress/payload-processing-5bf47cd988-6pshb"
Jul 23 22:01:51.140936 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.140907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njr8w\" (UniqueName: \"kubernetes.io/projected/f667c912-b23d-4b18-be7b-6f4574133175-kube-api-access-njr8w\") pod \"payload-processing-5bf47cd988-6pshb\" (UID: \"f667c912-b23d-4b18-be7b-6f4574133175\") " pod="openshift-ingress/payload-processing-5bf47cd988-6pshb"
Jul 23 22:01:51.163457 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.163421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:51.209742 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.209704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-6pshb"
Jul 23 22:01:51.306593 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.306504    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"]
Jul 23 22:01:51.310564 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:01:51.310534    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7886aac9_112c_4ad6_ae02_d7cff7963ce8.slice/crio-f363a30efde15723467c604ddb338df07606e3bf39166874021e1a5afbfdab31 WatchSource:0}: Error finding container f363a30efde15723467c604ddb338df07606e3bf39166874021e1a5afbfdab31: Status 404 returned error can't find the container with id f363a30efde15723467c604ddb338df07606e3bf39166874021e1a5afbfdab31
Jul 23 22:01:51.358998 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.358966    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-6pshb"]
Jul 23 22:01:51.361751 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:01:51.361728    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf667c912_b23d_4b18_be7b_6f4574133175.slice/crio-07a8ecd27fdc162a1c504d6d867f78738cb71edc6c06c8701ae6d9c8d406bd77 WatchSource:0}: Error finding container 07a8ecd27fdc162a1c504d6d867f78738cb71edc6c06c8701ae6d9c8d406bd77: Status 404 returned error can't find the container with id 07a8ecd27fdc162a1c504d6d867f78738cb71edc6c06c8701ae6d9c8d406bd77
Jul 23 22:01:51.452062 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.452018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q" event={"ID":"7886aac9-112c-4ad6-ae02-d7cff7963ce8","Type":"ContainerStarted","Data":"f363a30efde15723467c604ddb338df07606e3bf39166874021e1a5afbfdab31"}
Jul 23 22:01:51.453164 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:51.453139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-6pshb" event={"ID":"f667c912-b23d-4b18-be7b-6f4574133175","Type":"ContainerStarted","Data":"07a8ecd27fdc162a1c504d6d867f78738cb71edc6c06c8701ae6d9c8d406bd77"}
Jul 23 22:01:55.470220 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:55.470183    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q" event={"ID":"7886aac9-112c-4ad6-ae02-d7cff7963ce8","Type":"ContainerStarted","Data":"b0b5d943000628fa016826a3d72c2115a4afff2d60b4d95268ce24a087514540"}
Jul 23 22:01:55.470676 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:55.470253    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:01:55.471714 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:55.471689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-6pshb" event={"ID":"f667c912-b23d-4b18-be7b-6f4574133175","Type":"ContainerStarted","Data":"def7b3dad59b4c6463bf54f0930b3dec54caf861445d5768313f788a34b7ce8b"}
Jul 23 22:01:55.471789 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:55.471750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-6pshb"
Jul 23 22:01:55.487092 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:55.487022    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q" podStartSLOduration=2.124482267 podStartE2EDuration="5.487000805s" totalImagesPullingTime="3.362518538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:01:55.486316185 +0000 UTC m=+610.876708814" watchObservedRunningTime="2026-07-23 22:01:55.487000805 +0000 UTC m=+610.877393422"
Jul 23 22:01:55.502806 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:01:55.502743    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-6pshb" podStartSLOduration=2.217987308 podStartE2EDuration="5.502722627s" totalImagesPullingTime="3.284735319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:01:55.501648968 +0000 UTC m=+610.892041583" watchObservedRunningTime="2026-07-23 22:01:55.502722627 +0000 UTC m=+610.893115243"
Jul 23 22:02:01.481811 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:01.481781    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76bbd449ff-bf66q"
Jul 23 22:02:02.496388 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:02.496349    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:02:03.660037 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:03.659997    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:02:06.477312 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:06.477236    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-6pshb"
Jul 23 22:02:09.951234 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:09.951203    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:02:15.246533 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:15.246498    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:02:15.346153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:15.346117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:02:25.006825 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.006780    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"]
Jul 23 22:02:25.013884 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.013834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.016532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.016505    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 22:02:25.016679 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.016547    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-skzdc\""
Jul 23 22:02:25.016679 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.016560    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 22:02:25.017191 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.017174    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 23 22:02:25.020086 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.020057    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"]
Jul 23 22:02:25.038986 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.038949    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:02:25.118686 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.118649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5p4\" (UniqueName: \"kubernetes.io/projected/0a423e18-04e1-4b2f-9a2e-87653ce9214a-kube-api-access-8m5p4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.118686 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.118686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.118937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.118720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.118937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.118742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a423e18-04e1-4b2f-9a2e-87653ce9214a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.118937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.118777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.118937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.118848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.220218 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.220176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.220218 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.220224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.220479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.220245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a423e18-04e1-4b2f-9a2e-87653ce9214a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.220479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.220273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.220479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.220315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.220479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.220340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5p4\" (UniqueName: \"kubernetes.io/projected/0a423e18-04e1-4b2f-9a2e-87653ce9214a-kube-api-access-8m5p4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.220775 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.220657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.220775 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.220671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.220775 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.220725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.222822 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.222799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a423e18-04e1-4b2f-9a2e-87653ce9214a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.223062 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.223041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a423e18-04e1-4b2f-9a2e-87653ce9214a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.227684 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.227651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5p4\" (UniqueName: \"kubernetes.io/projected/0a423e18-04e1-4b2f-9a2e-87653ce9214a-kube-api-access-8m5p4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tnkf4\" (UID: \"0a423e18-04e1-4b2f-9a2e-87653ce9214a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.325965 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.325851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:25.460445 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.460417    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"]
Jul 23 22:02:25.462058 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:02:25.462027    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a423e18_04e1_4b2f_9a2e_87653ce9214a.slice/crio-6e78541e29e25e11a57b65dffb7379983d88dcbf0d362f23dba8bba2de372a0e WatchSource:0}: Error finding container 6e78541e29e25e11a57b65dffb7379983d88dcbf0d362f23dba8bba2de372a0e: Status 404 returned error can't find the container with id 6e78541e29e25e11a57b65dffb7379983d88dcbf0d362f23dba8bba2de372a0e
Jul 23 22:02:25.584978 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:25.584893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4" event={"ID":"0a423e18-04e1-4b2f-9a2e-87653ce9214a","Type":"ContainerStarted","Data":"6e78541e29e25e11a57b65dffb7379983d88dcbf0d362f23dba8bba2de372a0e"}
Jul 23 22:02:26.424025 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:26.423987    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:02:27.595767 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:27.595728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4" event={"ID":"0a423e18-04e1-4b2f-9a2e-87653ce9214a","Type":"ContainerStarted","Data":"f691400bf7bb941df069b2b542dd6b63c94d4c4c57c0f30fbd57aa36667d08eb"}
Jul 23 22:02:27.614736 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:27.614651    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4" podStartSLOduration=2.48000102 podStartE2EDuration="3.614633245s" totalImagesPullingTime="1.134632225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:02:27.612644173 +0000 UTC m=+643.003036790" watchObservedRunningTime="2026-07-23 22:02:27.614633245 +0000 UTC m=+643.005025860"
Jul 23 22:02:35.326292 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:35.326236    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:35.326292 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:35.326310    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:35.339071 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:35.339040    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:35.640340 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:35.640254    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tnkf4"
Jul 23 22:02:36.244529 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:36.244497    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:02:48.833082 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:48.833049    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:02:54.641733 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:54.641699    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:02:59.694597 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.694556    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"]
Jul 23 22:02:59.700747 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.700720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.702955 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.702926    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 23 22:02:59.708506 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.708474    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"]
Jul 23 22:02:59.725953 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.725917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.726126 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.725964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.726126 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.725989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87xc\" (UniqueName: \"kubernetes.io/projected/029169b6-1d2a-44cd-b9a9-a8d328300040-kube-api-access-l87xc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.726126 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.726023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029169b6-1d2a-44cd-b9a9-a8d328300040-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.726241 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.726186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.726241 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.726220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.729531 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.729496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:02:59.827558 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.827518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.827558 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.827557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.827783 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.827582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.827783 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.827606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.827783 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.827737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87xc\" (UniqueName: \"kubernetes.io/projected/029169b6-1d2a-44cd-b9a9-a8d328300040-kube-api-access-l87xc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.827975 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.827791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029169b6-1d2a-44cd-b9a9-a8d328300040-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.828028 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.827997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.828083 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.828063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.828140 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.828106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.830112 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.830088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/029169b6-1d2a-44cd-b9a9-a8d328300040-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.830342 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.830324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/029169b6-1d2a-44cd-b9a9-a8d328300040-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:02:59.836967 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:02:59.836944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87xc\" (UniqueName: \"kubernetes.io/projected/029169b6-1d2a-44cd-b9a9-a8d328300040-kube-api-access-l87xc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs\" (UID: \"029169b6-1d2a-44cd-b9a9-a8d328300040\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:03:00.010873 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:00.010830    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:03:00.149503 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:00.149467    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"]
Jul 23 22:03:00.152551 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:03:00.152523    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod029169b6_1d2a_44cd_b9a9_a8d328300040.slice/crio-d29b48a781fde346b91e6c21603c73cc2e4f61570adf29502ba8964aa182ef7a WatchSource:0}: Error finding container d29b48a781fde346b91e6c21603c73cc2e4f61570adf29502ba8964aa182ef7a: Status 404 returned error can't find the container with id d29b48a781fde346b91e6c21603c73cc2e4f61570adf29502ba8964aa182ef7a
Jul 23 22:03:00.153967 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:00.153949    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 22:03:00.733154 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:00.733120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs" event={"ID":"029169b6-1d2a-44cd-b9a9-a8d328300040","Type":"ContainerStarted","Data":"6b0a639df671c1820bf9cea3d7bdb534d284c61fed055bc319820f9436e7a7cf"}
Jul 23 22:03:00.733538 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:00.733162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs" event={"ID":"029169b6-1d2a-44cd-b9a9-a8d328300040","Type":"ContainerStarted","Data":"d29b48a781fde346b91e6c21603c73cc2e4f61570adf29502ba8964aa182ef7a"}
Jul 23 22:03:00.751548 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:00.751495    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs" podStartSLOduration=1.576598344 podStartE2EDuration="1.751477151s" totalImagesPullingTime="174.878807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:03:00.749591185 +0000 UTC m=+676.139983798" watchObservedRunningTime="2026-07-23 22:03:00.751477151 +0000 UTC m=+676.141869766"
Jul 23 22:03:10.011769 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:10.011727    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:03:10.012264 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:10.011791    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:03:10.024880 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:10.024836    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:03:10.784655 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:10.784624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs"
Jul 23 22:03:33.559054 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:33.559014    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:03:33.623020 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:03:33.622988    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:04:05.928841 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:05.928803    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:04:06.041202 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:06.041159    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:04:06.756501 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:06.756457    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68ffc9f8df-g57cd"]
Jul 23 22:04:06.756710 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:06.756656    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-68ffc9f8df-g57cd" podUID="97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2" containerName="authorino" containerID="cri-o://7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111" gracePeriod=30
Jul 23 22:04:07.003218 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.003191    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68ffc9f8df-g57cd"
Jul 23 22:04:07.005049 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.005018    2579 generic.go:356] "Generic (PLEG): container finished" podID="97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2" containerID="7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111" exitCode=0
Jul 23 22:04:07.005171 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.005066    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68ffc9f8df-g57cd"
Jul 23 22:04:07.005171 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.005093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68ffc9f8df-g57cd" event={"ID":"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2","Type":"ContainerDied","Data":"7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111"}
Jul 23 22:04:07.005171 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.005134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68ffc9f8df-g57cd" event={"ID":"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2","Type":"ContainerDied","Data":"fd305c58e732928d60525a5c255f34e2d8417d5d8f79ffa86d4f0ac4bc873317"}
Jul 23 22:04:07.005171 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.005151    2579 scope.go:117] "RemoveContainer" containerID="7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111"
Jul 23 22:04:07.013873 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.013838    2579 scope.go:117] "RemoveContainer" containerID="7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111"
Jul 23 22:04:07.014184 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:04:07.014162    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111\": container with ID starting with 7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111 not found: ID does not exist" containerID="7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111"
Jul 23 22:04:07.014254 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.014196    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111"} err="failed to get container status \"7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111\": rpc error: code = NotFound desc = could not find container \"7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111\": container with ID starting with 7220a5ac9b3ea6e703b40263adee2ab0e709f9ebfec2bc7872c59e07f024d111 not found: ID does not exist"
Jul 23 22:04:07.101982 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.101946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:04:07.113186 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.113160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgtxf\" (UniqueName: \"kubernetes.io/projected/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-kube-api-access-vgtxf\") pod \"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2\" (UID: \"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2\") "
Jul 23 22:04:07.113362 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.113209    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-tls-cert\") pod \"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2\" (UID: \"97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2\") "
Jul 23 22:04:07.115527 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.115461    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-kube-api-access-vgtxf" (OuterVolumeSpecName: "kube-api-access-vgtxf") pod "97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2" (UID: "97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2"). InnerVolumeSpecName "kube-api-access-vgtxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:04:07.122893 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.122845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2" (UID: "97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:04:07.196818 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.196779    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:04:07.214654 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.214623    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgtxf\" (UniqueName: \"kubernetes.io/projected/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-kube-api-access-vgtxf\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:04:07.214654 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.214655    2579 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2-tls-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:04:07.342898 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.341075    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68ffc9f8df-g57cd"]
Jul 23 22:04:07.346602 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:07.346577    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68ffc9f8df-g57cd"]
Jul 23 22:04:09.111653 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:04:09.111621    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2" path="/var/lib/kubelet/pods/97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2/volumes"
Jul 23 22:05:14.435165 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:05:14.435074    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:06:12.834690 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:06:12.834655    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:06:14.337881 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:06:14.337801    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:06:33.823496 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:06:33.823454    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:06:45.032158 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:06:45.032133    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:06:45.033175 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:06:45.033155    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:06:54.556123 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:06:54.556072    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:06:54.656479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:06:54.656429    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:06:54.751418 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:06:54.751383    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:07:00.900623 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:07:00.900578    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:07:00.972437 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:07:00.972401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:07:01.076622 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:07:01.076589    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:07:01.174179 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:07:01.174080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:07:01.312777 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:07:01.312735    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:07:07.269497 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:07:07.269462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:07:08.586274 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:07:08.586236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:08:19.578604 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:08:19.578521    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:11:45.056404 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:11:45.056375    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:11:45.056979 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:11:45.056779    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:12:07.868843 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:12:07.868805    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:12:07.952444 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:12:07.952398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:12:31.547771 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:12:31.547738    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:12:54.251690 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:12:54.251657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:15:00.148153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.148113    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747415-8r5jn"]
Jul 23 22:15:00.148595 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.148420    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2" containerName="authorino"
Jul 23 22:15:00.148595 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.148431    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2" containerName="authorino"
Jul 23 22:15:00.148595 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.148484    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="97d5577f-a446-4cbc-91f0-8fbf7a3cb4a2" containerName="authorino"
Jul 23 22:15:00.151438 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.151419    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-8r5jn"
Jul 23 22:15:00.154026 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.154000    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lq4mm\""
Jul 23 22:15:00.161126 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.161106    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-8r5jn"
Jul 23 22:15:00.168766 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.168738    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747415-8r5jn"]
Jul 23 22:15:00.301953 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.301927    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747415-8r5jn"]
Jul 23 22:15:00.304909 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:15:00.304874    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18c613f_f81d_4528_a09b_8036ddaa8b1c.slice/crio-12bc5cba24c6967e67f2a256b8b4e693ac67b89eaf8bc4a4798ee4b023846d6c WatchSource:0}: Error finding container 12bc5cba24c6967e67f2a256b8b4e693ac67b89eaf8bc4a4798ee4b023846d6c: Status 404 returned error can't find the container with id 12bc5cba24c6967e67f2a256b8b4e693ac67b89eaf8bc4a4798ee4b023846d6c
Jul 23 22:15:00.306748 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.306728    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 22:15:00.650192 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.650153    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-8r5jn" event={"ID":"d18c613f-f81d-4528-a09b-8036ddaa8b1c","Type":"ContainerStarted","Data":"12bc5cba24c6967e67f2a256b8b4e693ac67b89eaf8bc4a4798ee4b023846d6c"}
Jul 23 22:15:00.712720 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.712678    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"]
Jul 23 22:15:00.717327 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.717299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.719530 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.719499    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-bcc6bb8a-kserve-self-signed-certs\""
Jul 23 22:15:00.727231 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.727204    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"]
Jul 23 22:15:00.838380 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.838341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-model-cache\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.838563 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.838392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc84d69-2f46-4c01-a541-412da916b263-tls-certs\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.838563 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.838481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-dshm\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.838563 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.838532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-tmp-dir\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.838563 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.838561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-home\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.838702 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.838586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lllgl\" (UniqueName: \"kubernetes.io/projected/4bc84d69-2f46-4c01-a541-412da916b263-kube-api-access-lllgl\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.853576 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.853542    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:15:00.939404 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.939306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-model-cache\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.939404 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.939366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc84d69-2f46-4c01-a541-412da916b263-tls-certs\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.939648 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.939413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-dshm\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.939648 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.939437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-tmp-dir\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.939648 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.939584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-home\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.939807 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.939643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lllgl\" (UniqueName: \"kubernetes.io/projected/4bc84d69-2f46-4c01-a541-412da916b263-kube-api-access-lllgl\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.939807 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.939718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-model-cache\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.939807 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.939766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-tmp-dir\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.939999 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.939901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-home\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.941755 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.941732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-dshm\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.942052 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.942034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc84d69-2f46-4c01-a541-412da916b263-tls-certs\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:00.947352 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:00.947329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lllgl\" (UniqueName: \"kubernetes.io/projected/4bc84d69-2f46-4c01-a541-412da916b263-kube-api-access-lllgl\") pod \"e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") " pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:01.028128 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:01.028085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:01.163842 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:01.163806    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"]
Jul 23 22:15:01.167201 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:15:01.167165    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bc84d69_2f46_4c01_a541_412da916b263.slice/crio-a5da53bb886ad957553fcb8d630dbb9dd20d7264c6c4da09c26d01ecefd63847 WatchSource:0}: Error finding container a5da53bb886ad957553fcb8d630dbb9dd20d7264c6c4da09c26d01ecefd63847: Status 404 returned error can't find the container with id a5da53bb886ad957553fcb8d630dbb9dd20d7264c6c4da09c26d01ecefd63847
Jul 23 22:15:01.657725 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:01.657625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4" event={"ID":"4bc84d69-2f46-4c01-a541-412da916b263","Type":"ContainerStarted","Data":"2956c4a033ce0717589dfbc53f48ee3f3f7c9e40133a2893fcf75d80fb2d3111"}
Jul 23 22:15:01.657725 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:01.657675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4" event={"ID":"4bc84d69-2f46-4c01-a541-412da916b263","Type":"ContainerStarted","Data":"a5da53bb886ad957553fcb8d630dbb9dd20d7264c6c4da09c26d01ecefd63847"}
Jul 23 22:15:01.675565 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:01.675515    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4" podStartSLOduration=1.6754975349999999 podStartE2EDuration="1.675497535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:15:01.674132246 +0000 UTC m=+1397.064524864" watchObservedRunningTime="2026-07-23 22:15:01.675497535 +0000 UTC m=+1397.065890150"
Jul 23 22:15:03.666715 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:03.666680    2579 generic.go:356] "Generic (PLEG): container finished" podID="d18c613f-f81d-4528-a09b-8036ddaa8b1c" containerID="b1005e55c586914e074136a1d9ce96891bb449cacc2c7d323440756809afa0f7" exitCode=0
Jul 23 22:15:03.667140 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:03.666753    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-8r5jn" event={"ID":"d18c613f-f81d-4528-a09b-8036ddaa8b1c","Type":"ContainerDied","Data":"b1005e55c586914e074136a1d9ce96891bb449cacc2c7d323440756809afa0f7"}
Jul 23 22:15:04.805701 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:04.805670    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-8r5jn"
Jul 23 22:15:05.676590 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:05.676557    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-8r5jn"
Jul 23 22:15:05.676590 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:05.676567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747415-8r5jn" event={"ID":"d18c613f-f81d-4528-a09b-8036ddaa8b1c","Type":"ContainerDied","Data":"12bc5cba24c6967e67f2a256b8b4e693ac67b89eaf8bc4a4798ee4b023846d6c"}
Jul 23 22:15:05.676818 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:05.676613    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12bc5cba24c6967e67f2a256b8b4e693ac67b89eaf8bc4a4798ee4b023846d6c"
Jul 23 22:15:09.812142 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.812104    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"]
Jul 23 22:15:09.812619 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.812399    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d18c613f-f81d-4528-a09b-8036ddaa8b1c" containerName="cleanup"
Jul 23 22:15:09.812619 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.812410    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d18c613f-f81d-4528-a09b-8036ddaa8b1c" containerName="cleanup"
Jul 23 22:15:09.812619 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.812463    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d18c613f-f81d-4528-a09b-8036ddaa8b1c" containerName="cleanup"
Jul 23 22:15:09.815075 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.815056    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:09.817507 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.817483    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-bcc6bb8a-kserve-self-signed-certs\""
Jul 23 22:15:09.826583 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.826553    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"]
Jul 23 22:15:09.842195 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.842166    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:15:09.923097 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.923058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-tmp-dir\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:09.923298 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.923119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-dshm\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:09.923298 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.923135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-model-cache\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:09.923298 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.923174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-home\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:09.923298 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.923189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7112e3c1-d0cc-417a-aca8-c92451200ca2-tls-certs\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:09.923298 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:09.923224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njz2p\" (UniqueName: \"kubernetes.io/projected/7112e3c1-d0cc-417a-aca8-c92451200ca2-kube-api-access-njz2p\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.024579 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.024548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-dshm\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.024579 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.024581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-model-cache\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.024824 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.024610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-home\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.024824 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.024627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7112e3c1-d0cc-417a-aca8-c92451200ca2-tls-certs\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.024824 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.024672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njz2p\" (UniqueName: \"kubernetes.io/projected/7112e3c1-d0cc-417a-aca8-c92451200ca2-kube-api-access-njz2p\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.024824 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.024724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-tmp-dir\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.025114 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.025092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-model-cache\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.025179 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.025116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-home\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.025179 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.025142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-tmp-dir\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.027010 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.026989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-dshm\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.027185 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.027166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7112e3c1-d0cc-417a-aca8-c92451200ca2-tls-certs\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.034888 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.034837    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njz2p\" (UniqueName: \"kubernetes.io/projected/7112e3c1-d0cc-417a-aca8-c92451200ca2-kube-api-access-njz2p\") pod \"e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") " pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.124384 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.124290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:10.261462 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.261437    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"]
Jul 23 22:15:10.263736 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:15:10.263707    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7112e3c1_d0cc_417a_aca8_c92451200ca2.slice/crio-e295ea770991bb8c21209b16f19f77174bd4355ad0d68b7ac723e81b0c9d971e WatchSource:0}: Error finding container e295ea770991bb8c21209b16f19f77174bd4355ad0d68b7ac723e81b0c9d971e: Status 404 returned error can't find the container with id e295ea770991bb8c21209b16f19f77174bd4355ad0d68b7ac723e81b0c9d971e
Jul 23 22:15:10.698922 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.698886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz" event={"ID":"7112e3c1-d0cc-417a-aca8-c92451200ca2","Type":"ContainerStarted","Data":"69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46"}
Jul 23 22:15:10.698922 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.698931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz" event={"ID":"7112e3c1-d0cc-417a-aca8-c92451200ca2","Type":"ContainerStarted","Data":"e295ea770991bb8c21209b16f19f77174bd4355ad0d68b7ac723e81b0c9d971e"}
Jul 23 22:15:10.726425 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:10.726354    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz" podStartSLOduration=1.726337289 podStartE2EDuration="1.726337289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:15:10.726110151 +0000 UTC m=+1406.116502768" watchObservedRunningTime="2026-07-23 22:15:10.726337289 +0000 UTC m=+1406.116729904"
Jul 23 22:15:11.028609 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:11.028567    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:11.029091 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:11.028622    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:11.043144 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:11.043112    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:11.714686 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:11.714658    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:15:20.124705 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:20.124666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:20.125235 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:20.124722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:20.141814 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:20.141784    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:15:20.752458 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:15:20.752430    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:16:00.752914 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:00.752844    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"]
Jul 23 22:16:00.776938 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:00.776905    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:16:02.423519 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:02.423484    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"]
Jul 23 22:16:02.491284 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:02.491256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:16:15.773379 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:15.773329    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4" podUID="4bc84d69-2f46-4c01-a541-412da916b263" containerName="main" containerID="cri-o://2956c4a033ce0717589dfbc53f48ee3f3f7c9e40133a2893fcf75d80fb2d3111" gracePeriod=45
Jul 23 22:16:15.975605 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:15.975561    2579 generic.go:356] "Generic (PLEG): container finished" podID="4bc84d69-2f46-4c01-a541-412da916b263" containerID="2956c4a033ce0717589dfbc53f48ee3f3f7c9e40133a2893fcf75d80fb2d3111" exitCode=0
Jul 23 22:16:15.975754 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:15.975626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4" event={"ID":"4bc84d69-2f46-4c01-a541-412da916b263","Type":"ContainerDied","Data":"2956c4a033ce0717589dfbc53f48ee3f3f7c9e40133a2893fcf75d80fb2d3111"}
Jul 23 22:16:16.021952 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.021925    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:16:16.099026 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.098928    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lllgl\" (UniqueName: \"kubernetes.io/projected/4bc84d69-2f46-4c01-a541-412da916b263-kube-api-access-lllgl\") pod \"4bc84d69-2f46-4c01-a541-412da916b263\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") "
Jul 23 22:16:16.099026 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.098989    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc84d69-2f46-4c01-a541-412da916b263-tls-certs\") pod \"4bc84d69-2f46-4c01-a541-412da916b263\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") "
Jul 23 22:16:16.099238 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.099110    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-home\") pod \"4bc84d69-2f46-4c01-a541-412da916b263\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") "
Jul 23 22:16:16.099238 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.099151    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-model-cache\") pod \"4bc84d69-2f46-4c01-a541-412da916b263\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") "
Jul 23 22:16:16.099238 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.099178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-tmp-dir\") pod \"4bc84d69-2f46-4c01-a541-412da916b263\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") "
Jul 23 22:16:16.099238 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.099215    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-dshm\") pod \"4bc84d69-2f46-4c01-a541-412da916b263\" (UID: \"4bc84d69-2f46-4c01-a541-412da916b263\") "
Jul 23 22:16:16.099442 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.099420    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-model-cache" (OuterVolumeSpecName: "model-cache") pod "4bc84d69-2f46-4c01-a541-412da916b263" (UID: "4bc84d69-2f46-4c01-a541-412da916b263"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:16:16.099493 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.099424    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-home" (OuterVolumeSpecName: "home") pod "4bc84d69-2f46-4c01-a541-412da916b263" (UID: "4bc84d69-2f46-4c01-a541-412da916b263"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:16:16.099493 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.099458    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4bc84d69-2f46-4c01-a541-412da916b263" (UID: "4bc84d69-2f46-4c01-a541-412da916b263"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:16:16.101396 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.101369    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc84d69-2f46-4c01-a541-412da916b263-kube-api-access-lllgl" (OuterVolumeSpecName: "kube-api-access-lllgl") pod "4bc84d69-2f46-4c01-a541-412da916b263" (UID: "4bc84d69-2f46-4c01-a541-412da916b263"). InnerVolumeSpecName "kube-api-access-lllgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:16:16.101396 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.101383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-dshm" (OuterVolumeSpecName: "dshm") pod "4bc84d69-2f46-4c01-a541-412da916b263" (UID: "4bc84d69-2f46-4c01-a541-412da916b263"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:16:16.101548 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.101412    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bc84d69-2f46-4c01-a541-412da916b263-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4bc84d69-2f46-4c01-a541-412da916b263" (UID: "4bc84d69-2f46-4c01-a541-412da916b263"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:16:16.199855 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.199812    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:16.199855 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.199848    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lllgl\" (UniqueName: \"kubernetes.io/projected/4bc84d69-2f46-4c01-a541-412da916b263-kube-api-access-lllgl\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:16.199855 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.199885    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc84d69-2f46-4c01-a541-412da916b263-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:16.200131 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.199895    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:16.200131 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.199904    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:16.200131 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.199911    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc84d69-2f46-4c01-a541-412da916b263-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:16.980945 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.980904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4" event={"ID":"4bc84d69-2f46-4c01-a541-412da916b263","Type":"ContainerDied","Data":"a5da53bb886ad957553fcb8d630dbb9dd20d7264c6c4da09c26d01ecefd63847"}
Jul 23 22:16:16.981361 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.980924    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"
Jul 23 22:16:16.981361 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:16.980961    2579 scope.go:117] "RemoveContainer" containerID="2956c4a033ce0717589dfbc53f48ee3f3f7c9e40133a2893fcf75d80fb2d3111"
Jul 23 22:16:17.005812 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.005778    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"]
Jul 23 22:16:17.012452 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.012421    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-bcc6bb8a-kserve-55bc64ff47-2rdp4"]
Jul 23 22:16:17.111685 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.111650    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bc84d69-2f46-4c01-a541-412da916b263" path="/var/lib/kubelet/pods/4bc84d69-2f46-4c01-a541-412da916b263/volumes"
Jul 23 22:16:17.444063 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.443951    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz" podUID="7112e3c1-d0cc-417a-aca8-c92451200ca2" containerName="main" containerID="cri-o://69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46" gracePeriod=45
Jul 23 22:16:17.702434 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.702412    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:16:17.712469 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.712444    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-tmp-dir\") pod \"7112e3c1-d0cc-417a-aca8-c92451200ca2\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") "
Jul 23 22:16:17.712565 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.712477    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njz2p\" (UniqueName: \"kubernetes.io/projected/7112e3c1-d0cc-417a-aca8-c92451200ca2-kube-api-access-njz2p\") pod \"7112e3c1-d0cc-417a-aca8-c92451200ca2\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") "
Jul 23 22:16:17.712565 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.712507    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-dshm\") pod \"7112e3c1-d0cc-417a-aca8-c92451200ca2\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") "
Jul 23 22:16:17.712565 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.712538    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-model-cache\") pod \"7112e3c1-d0cc-417a-aca8-c92451200ca2\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") "
Jul 23 22:16:17.712721 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.712577    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7112e3c1-d0cc-417a-aca8-c92451200ca2-tls-certs\") pod \"7112e3c1-d0cc-417a-aca8-c92451200ca2\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") "
Jul 23 22:16:17.712721 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.712596    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-home\") pod \"7112e3c1-d0cc-417a-aca8-c92451200ca2\" (UID: \"7112e3c1-d0cc-417a-aca8-c92451200ca2\") "
Jul 23 22:16:17.712823 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.712717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7112e3c1-d0cc-417a-aca8-c92451200ca2" (UID: "7112e3c1-d0cc-417a-aca8-c92451200ca2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:16:17.712823 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.712799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-model-cache" (OuterVolumeSpecName: "model-cache") pod "7112e3c1-d0cc-417a-aca8-c92451200ca2" (UID: "7112e3c1-d0cc-417a-aca8-c92451200ca2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:16:17.713023 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.712916    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:17.713023 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.712919    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-home" (OuterVolumeSpecName: "home") pod "7112e3c1-d0cc-417a-aca8-c92451200ca2" (UID: "7112e3c1-d0cc-417a-aca8-c92451200ca2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:16:17.714752 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.714731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-dshm" (OuterVolumeSpecName: "dshm") pod "7112e3c1-d0cc-417a-aca8-c92451200ca2" (UID: "7112e3c1-d0cc-417a-aca8-c92451200ca2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:16:17.714805 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.714764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7112e3c1-d0cc-417a-aca8-c92451200ca2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7112e3c1-d0cc-417a-aca8-c92451200ca2" (UID: "7112e3c1-d0cc-417a-aca8-c92451200ca2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:16:17.714851 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.714835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7112e3c1-d0cc-417a-aca8-c92451200ca2-kube-api-access-njz2p" (OuterVolumeSpecName: "kube-api-access-njz2p") pod "7112e3c1-d0cc-417a-aca8-c92451200ca2" (UID: "7112e3c1-d0cc-417a-aca8-c92451200ca2"). InnerVolumeSpecName "kube-api-access-njz2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:16:17.813737 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.813706    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njz2p\" (UniqueName: \"kubernetes.io/projected/7112e3c1-d0cc-417a-aca8-c92451200ca2-kube-api-access-njz2p\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:17.813737 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.813735    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:17.813737 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.813744    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:17.814022 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.813753    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7112e3c1-d0cc-417a-aca8-c92451200ca2-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:17.814022 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.813761    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7112e3c1-d0cc-417a-aca8-c92451200ca2-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:16:17.986736 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.986642    2579 generic.go:356] "Generic (PLEG): container finished" podID="7112e3c1-d0cc-417a-aca8-c92451200ca2" containerID="69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46" exitCode=0
Jul 23 22:16:17.986736 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.986684    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"
Jul 23 22:16:17.987247 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.986731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz" event={"ID":"7112e3c1-d0cc-417a-aca8-c92451200ca2","Type":"ContainerDied","Data":"69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46"}
Jul 23 22:16:17.987247 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.986772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz" event={"ID":"7112e3c1-d0cc-417a-aca8-c92451200ca2","Type":"ContainerDied","Data":"e295ea770991bb8c21209b16f19f77174bd4355ad0d68b7ac723e81b0c9d971e"}
Jul 23 22:16:17.987247 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.986789    2579 scope.go:117] "RemoveContainer" containerID="69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46"
Jul 23 22:16:17.995228 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.995213    2579 scope.go:117] "RemoveContainer" containerID="69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46"
Jul 23 22:16:17.995507 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:16:17.995487    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46\": container with ID starting with 69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46 not found: ID does not exist" containerID="69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46"
Jul 23 22:16:17.995588 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:17.995519    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46"} err="failed to get container status \"69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46\": rpc error: code = NotFound desc = could not find container \"69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46\": container with ID starting with 69476cdcc70dd958845cb7f64a570dba97d0d8688ad7a418550626809c035a46 not found: ID does not exist"
Jul 23 22:16:18.008685 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:18.008653    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"]
Jul 23 22:16:18.012772 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:18.012748    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-bcc6bb8a-kserve-b5765c695-x57hz"]
Jul 23 22:16:19.111472 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:19.111434    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7112e3c1-d0cc-417a-aca8-c92451200ca2" path="/var/lib/kubelet/pods/7112e3c1-d0cc-417a-aca8-c92451200ca2/volumes"
Jul 23 22:16:45.081115 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:45.081081    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:16:45.081605 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:16:45.081200    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:18:38.555790 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:18:38.555711    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:14.563149 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:14.563114    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:38.272948 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:38.272910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:41.470801 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:41.470766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:41.565780 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:41.565746    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:41.934987 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:41.934910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:47.843033 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:47.842993    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:47.910525 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:47.910495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:48.085941 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:48.085905    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:48.412875 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:48.412838    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:48.812880 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:48.812819    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:50.538323 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.538286    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:50.618232 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.618191    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"]
Jul 23 22:19:50.618530 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.618517    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bc84d69-2f46-4c01-a541-412da916b263" containerName="main"
Jul 23 22:19:50.618577 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.618532    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bc84d69-2f46-4c01-a541-412da916b263" containerName="main"
Jul 23 22:19:50.618577 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.618543    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7112e3c1-d0cc-417a-aca8-c92451200ca2" containerName="main"
Jul 23 22:19:50.618577 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.618548    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7112e3c1-d0cc-417a-aca8-c92451200ca2" containerName="main"
Jul 23 22:19:50.618683 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.618603    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7112e3c1-d0cc-417a-aca8-c92451200ca2" containerName="main"
Jul 23 22:19:50.618683 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.618611    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bc84d69-2f46-4c01-a541-412da916b263" containerName="main"
Jul 23 22:19:50.621538 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.621513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:50.623442 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.623415    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:50.623981 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.623958    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-487a121b-serving-cert\""
Jul 23 22:19:50.631838 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.631815    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"]
Jul 23 22:19:50.698154 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.698116    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:50.776401 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.776366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4a7c9c-d491-43b8-9411-8208072c3aff-maas-api-tls\") pod \"maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:50.776601 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.776407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a7c9c-d491-43b8-9411-8208072c3aff-openshift-service-ca\") pod \"maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:50.776601 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.776431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw2hn\" (UniqueName: \"kubernetes.io/projected/8f4a7c9c-d491-43b8-9411-8208072c3aff-kube-api-access-qw2hn\") pod \"maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:50.877623 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.877541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4a7c9c-d491-43b8-9411-8208072c3aff-maas-api-tls\") pod \"maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:50.877623 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.877591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a7c9c-d491-43b8-9411-8208072c3aff-openshift-service-ca\") pod \"maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:50.877623 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.877619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw2hn\" (UniqueName: \"kubernetes.io/projected/8f4a7c9c-d491-43b8-9411-8208072c3aff-kube-api-access-qw2hn\") pod \"maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:50.877936 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:19:50.877695    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-serving-cert: secret "maas-api-e2e-ait-487a121b-serving-cert" not found
Jul 23 22:19:50.877936 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:19:50.877763    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f4a7c9c-d491-43b8-9411-8208072c3aff-maas-api-tls podName:8f4a7c9c-d491-43b8-9411-8208072c3aff nodeName:}" failed. No retries permitted until 2026-07-23 22:19:51.377747502 +0000 UTC m=+1686.768140096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f4a7c9c-d491-43b8-9411-8208072c3aff-maas-api-tls") pod "maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw" (UID: "8f4a7c9c-d491-43b8-9411-8208072c3aff") : secret "maas-api-e2e-ait-487a121b-serving-cert" not found
Jul 23 22:19:50.878252 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.878232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a7c9c-d491-43b8-9411-8208072c3aff-openshift-service-ca\") pod \"maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:50.887449 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.887421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw2hn\" (UniqueName: \"kubernetes.io/projected/8f4a7c9c-d491-43b8-9411-8208072c3aff-kube-api-access-qw2hn\") pod \"maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:50.973204 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:50.973173    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:51.364508 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:51.364455    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:51.382437 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:51.382396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4a7c9c-d491-43b8-9411-8208072c3aff-maas-api-tls\") pod \"maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:51.385036 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:51.385003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4a7c9c-d491-43b8-9411-8208072c3aff-maas-api-tls\") pod \"maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:51.461150 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:51.461115    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:19:51.533292 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:51.533247    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:51.669786 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:51.669758    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"]
Jul 23 22:19:51.672398 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:19:51.672355    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4a7c9c_d491_43b8_9411_8208072c3aff.slice/crio-c02166dcb67d342033aaaf23fd5562110a87847aa536c87a44b0b8eb6e7ccb62 WatchSource:0}: Error finding container c02166dcb67d342033aaaf23fd5562110a87847aa536c87a44b0b8eb6e7ccb62: Status 404 returned error can't find the container with id c02166dcb67d342033aaaf23fd5562110a87847aa536c87a44b0b8eb6e7ccb62
Jul 23 22:19:51.889042 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:51.888948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw" event={"ID":"8f4a7c9c-d491-43b8-9411-8208072c3aff","Type":"ContainerStarted","Data":"c02166dcb67d342033aaaf23fd5562110a87847aa536c87a44b0b8eb6e7ccb62"}
Jul 23 22:19:52.895294 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:52.895262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw" event={"ID":"8f4a7c9c-d491-43b8-9411-8208072c3aff","Type":"ContainerStarted","Data":"d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474"}
Jul 23 22:19:52.895676 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:52.895383    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:19:52.915724 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:52.915663    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw" podStartSLOduration=2.484615197 podStartE2EDuration="2.915642568s" totalImagesPullingTime="431.027371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:19:52.91374474 +0000 UTC m=+1688.304137358" watchObservedRunningTime="2026-07-23 22:19:52.915642568 +0000 UTC m=+1688.306035184"
Jul 23 22:19:58.905479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:19:58.905450    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:20:16.174525 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:16.174435    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"]
Jul 23 22:20:16.177772 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:16.177747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"
Jul 23 22:20:16.180143 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:16.180122    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lq4mm\""
Jul 23 22:20:16.186363 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:16.186334    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"]
Jul 23 22:20:16.208286 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:16.208253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e83fe6-005b-489e-830d-7e5774fd5950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-487a121b-rqv5h\" (UID: \"e0e83fe6-005b-489e-830d-7e5774fd5950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"
Jul 23 22:20:16.308967 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:16.308920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e83fe6-005b-489e-830d-7e5774fd5950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-487a121b-rqv5h\" (UID: \"e0e83fe6-005b-489e-830d-7e5774fd5950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"
Jul 23 22:20:16.309548 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:16.309526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e83fe6-005b-489e-830d-7e5774fd5950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-487a121b-rqv5h\" (UID: \"e0e83fe6-005b-489e-830d-7e5774fd5950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"
Jul 23 22:20:16.489260 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:16.489227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"
Jul 23 22:20:16.623298 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:16.623266    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"]
Jul 23 22:20:16.626534 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:20:16.626507    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e83fe6_005b_489e_830d_7e5774fd5950.slice/crio-f111a996b1c6053bff3177d64a81bcfad6c4a762b20192b8f63eaad319b5ff1c WatchSource:0}: Error finding container f111a996b1c6053bff3177d64a81bcfad6c4a762b20192b8f63eaad319b5ff1c: Status 404 returned error can't find the container with id f111a996b1c6053bff3177d64a81bcfad6c4a762b20192b8f63eaad319b5ff1c
Jul 23 22:20:16.995343 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:16.995309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h" event={"ID":"e0e83fe6-005b-489e-830d-7e5774fd5950","Type":"ContainerStarted","Data":"282832f698c3aefcbeb3092d995fa533896fa2a7740c2b946aea1a9e2d1f374d"}
Jul 23 22:20:16.995525 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:16.995363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h" event={"ID":"e0e83fe6-005b-489e-830d-7e5774fd5950","Type":"ContainerStarted","Data":"f111a996b1c6053bff3177d64a81bcfad6c4a762b20192b8f63eaad319b5ff1c"}
Jul 23 22:20:17.012349 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:17.012300    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h" podStartSLOduration=1.012281918 podStartE2EDuration="1.012281918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:20:17.010767475 +0000 UTC m=+1712.401160092" watchObservedRunningTime="2026-07-23 22:20:17.012281918 +0000 UTC m=+1712.402674534"
Jul 23 22:20:18.000621 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:18.000587    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0e83fe6-005b-489e-830d-7e5774fd5950" containerID="282832f698c3aefcbeb3092d995fa533896fa2a7740c2b946aea1a9e2d1f374d" exitCode=0
Jul 23 22:20:18.001058 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:18.000675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h" event={"ID":"e0e83fe6-005b-489e-830d-7e5774fd5950","Type":"ContainerDied","Data":"282832f698c3aefcbeb3092d995fa533896fa2a7740c2b946aea1a9e2d1f374d"}
Jul 23 22:20:19.134577 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:19.134551    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"
Jul 23 22:20:19.232225 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:19.232180    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e83fe6-005b-489e-830d-7e5774fd5950-maas-api-service-ca\") pod \"e0e83fe6-005b-489e-830d-7e5774fd5950\" (UID: \"e0e83fe6-005b-489e-830d-7e5774fd5950\") "
Jul 23 22:20:19.232585 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:19.232564    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e83fe6-005b-489e-830d-7e5774fd5950-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0e83fe6-005b-489e-830d-7e5774fd5950" (UID: "e0e83fe6-005b-489e-830d-7e5774fd5950"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:20:19.333574 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:19.333479    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0e83fe6-005b-489e-830d-7e5774fd5950-maas-api-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:20:20.010700 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:20.010671    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"
Jul 23 22:20:20.010903 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:20.010701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h" event={"ID":"e0e83fe6-005b-489e-830d-7e5774fd5950","Type":"ContainerDied","Data":"f111a996b1c6053bff3177d64a81bcfad6c4a762b20192b8f63eaad319b5ff1c"}
Jul 23 22:20:20.010903 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:20.010757    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f111a996b1c6053bff3177d64a81bcfad6c4a762b20192b8f63eaad319b5ff1c"
Jul 23 22:20:26.635114 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:26.635073    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"]
Jul 23 22:20:26.635652 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:26.635377    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw" podUID="8f4a7c9c-d491-43b8-9411-8208072c3aff" containerName="maas-api" containerID="cri-o://d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474" gracePeriod=30
Jul 23 22:20:26.666908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:26.666852    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:26.877668 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:26.877640    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:20:27.005842 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.005809    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a7c9c-d491-43b8-9411-8208072c3aff-openshift-service-ca\") pod \"8f4a7c9c-d491-43b8-9411-8208072c3aff\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") "
Jul 23 22:20:27.006032 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.005926    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw2hn\" (UniqueName: \"kubernetes.io/projected/8f4a7c9c-d491-43b8-9411-8208072c3aff-kube-api-access-qw2hn\") pod \"8f4a7c9c-d491-43b8-9411-8208072c3aff\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") "
Jul 23 22:20:27.006032 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.005979    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4a7c9c-d491-43b8-9411-8208072c3aff-maas-api-tls\") pod \"8f4a7c9c-d491-43b8-9411-8208072c3aff\" (UID: \"8f4a7c9c-d491-43b8-9411-8208072c3aff\") "
Jul 23 22:20:27.006241 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.006222    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4a7c9c-d491-43b8-9411-8208072c3aff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f4a7c9c-d491-43b8-9411-8208072c3aff" (UID: "8f4a7c9c-d491-43b8-9411-8208072c3aff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:20:27.008315 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.008274    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f4a7c9c-d491-43b8-9411-8208072c3aff-kube-api-access-qw2hn" (OuterVolumeSpecName: "kube-api-access-qw2hn") pod "8f4a7c9c-d491-43b8-9411-8208072c3aff" (UID: "8f4a7c9c-d491-43b8-9411-8208072c3aff"). InnerVolumeSpecName "kube-api-access-qw2hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:20:27.008315 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.008297    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f4a7c9c-d491-43b8-9411-8208072c3aff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f4a7c9c-d491-43b8-9411-8208072c3aff" (UID: "8f4a7c9c-d491-43b8-9411-8208072c3aff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:20:27.044316 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.044274    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f4a7c9c-d491-43b8-9411-8208072c3aff" containerID="d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474" exitCode=0
Jul 23 22:20:27.044510 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.044322    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"
Jul 23 22:20:27.044510 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.044350    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw" event={"ID":"8f4a7c9c-d491-43b8-9411-8208072c3aff","Type":"ContainerDied","Data":"d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474"}
Jul 23 22:20:27.044510 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.044398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw" event={"ID":"8f4a7c9c-d491-43b8-9411-8208072c3aff","Type":"ContainerDied","Data":"c02166dcb67d342033aaaf23fd5562110a87847aa536c87a44b0b8eb6e7ccb62"}
Jul 23 22:20:27.044510 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.044423    2579 scope.go:117] "RemoveContainer" containerID="d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474"
Jul 23 22:20:27.053487 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.053460    2579 scope.go:117] "RemoveContainer" containerID="d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474"
Jul 23 22:20:27.053733 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:20:27.053708    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474\": container with ID starting with d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474 not found: ID does not exist" containerID="d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474"
Jul 23 22:20:27.053813 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.053746    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474"} err="failed to get container status \"d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474\": rpc error: code = NotFound desc = could not find container \"d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474\": container with ID starting with d42d96eed4a852e5599ff2fd79cdffcfa3236616012e77d90eebb4938a697474 not found: ID does not exist"
Jul 23 22:20:27.066498 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.066470    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"]
Jul 23 22:20:27.070811 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.070783    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-487a121b-7f9b7c596d-cmbtw"]
Jul 23 22:20:27.106565 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.106528    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4a7c9c-d491-43b8-9411-8208072c3aff-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:20:27.106565 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.106561    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw2hn\" (UniqueName: \"kubernetes.io/projected/8f4a7c9c-d491-43b8-9411-8208072c3aff-kube-api-access-qw2hn\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:20:27.106565 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.106573    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f4a7c9c-d491-43b8-9411-8208072c3aff-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:20:27.111483 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:27.111455    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f4a7c9c-d491-43b8-9411-8208072c3aff" path="/var/lib/kubelet/pods/8f4a7c9c-d491-43b8-9411-8208072c3aff/volumes"
Jul 23 22:20:31.279048 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:31.279000    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:31.280560 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:31.280541    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"]
Jul 23 22:20:31.285979 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:31.285951    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-487a121b-rqv5h"]
Jul 23 22:20:31.436808 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:31.436775    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:31.688166 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:31.688081    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:32.008443 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.008405    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:32.152822 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.152789    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:32.253079 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.253049    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:32.493593 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.493556    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"]
Jul 23 22:20:32.494027 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.493874    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0e83fe6-005b-489e-830d-7e5774fd5950" containerName="revoke-keys"
Jul 23 22:20:32.494027 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.493888    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0e83fe6-005b-489e-830d-7e5774fd5950" containerName="revoke-keys"
Jul 23 22:20:32.494027 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.493905    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4a7c9c-d491-43b8-9411-8208072c3aff" containerName="maas-api"
Jul 23 22:20:32.494027 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.493911    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4a7c9c-d491-43b8-9411-8208072c3aff" containerName="maas-api"
Jul 23 22:20:32.494027 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.493958    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0e83fe6-005b-489e-830d-7e5774fd5950" containerName="revoke-keys"
Jul 23 22:20:32.494027 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.493968    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4a7c9c-d491-43b8-9411-8208072c3aff" containerName="maas-api"
Jul 23 22:20:32.500278 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.500251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.502604 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.502581    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 22:20:32.502738 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.502640    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-58ac7283-gw-openshift-default-dockercfg-kh5sm\""
Jul 23 22:20:32.511196 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.511165    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"]
Jul 23 22:20:32.515590 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.515563    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:32.648484 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.648447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:32.658708 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.658668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-token\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.658708 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.658719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-podinfo\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.658983 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.658792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsz8j\" (UniqueName: \"kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-kube-api-access-rsz8j\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.658983 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.658899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-certs\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.658983 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.658925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-data\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.659108 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.659002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-socket\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.659108 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.659025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istiod-ca-cert\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.659108 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.659043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-credential-socket\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.659108 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.659071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-envoy\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.759772 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.759678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-socket\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.759772 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.759718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istiod-ca-cert\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.759772 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.759741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-credential-socket\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.759772 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.759760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-envoy\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.759772 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.759776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-token\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.760229 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.759796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-podinfo\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.760229 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.759838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsz8j\" (UniqueName: \"kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-kube-api-access-rsz8j\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.760229 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.759900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-certs\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.760229 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.759926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-data\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.760229 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.760182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-socket\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.760491 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.760265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-credential-socket\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.760491 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.760275    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-data\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.760491 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.760434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-certs\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.760788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.760763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istiod-ca-cert\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.762245 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.762221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-envoy\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.762445 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.762427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-podinfo\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.768677 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.768650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsz8j\" (UniqueName: \"kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-kube-api-access-rsz8j\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.769016 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.768994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-token\") pod \"e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") " pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.812054 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.812005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:32.954709 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.954684    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"]
Jul 23 22:20:32.956712 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:20:32.956683    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e19d92_e199_4071_9dc8_c29d2abb2bdd.slice/crio-111e15e6e00904eb9fee1599696c8f30b5eb1cc89420e6e34ad80b64534f9a6c WatchSource:0}: Error finding container 111e15e6e00904eb9fee1599696c8f30b5eb1cc89420e6e34ad80b64534f9a6c: Status 404 returned error can't find the container with id 111e15e6e00904eb9fee1599696c8f30b5eb1cc89420e6e34ad80b64534f9a6c
Jul 23 22:20:32.958748 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:32.958729    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 22:20:33.071506 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:33.071422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" event={"ID":"e7e19d92-e199-4071-9dc8-c29d2abb2bdd","Type":"ContainerStarted","Data":"111e15e6e00904eb9fee1599696c8f30b5eb1cc89420e6e34ad80b64534f9a6c"}
Jul 23 22:20:33.112092 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:33.112062    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0e83fe6-005b-489e-830d-7e5774fd5950" path="/var/lib/kubelet/pods/e0e83fe6-005b-489e-830d-7e5774fd5950/volumes"
Jul 23 22:20:35.870267 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:35.870223    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:20:35.870641 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:35.870315    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:20:35.870641 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:35.870384    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:20:36.088414 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:36.088378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" event={"ID":"e7e19d92-e199-4071-9dc8-c29d2abb2bdd","Type":"ContainerStarted","Data":"ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d"}
Jul 23 22:20:36.111820 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:36.111761    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" podStartSLOduration=1.2007509490000001 podStartE2EDuration="4.111740423s" totalImagesPullingTime="2.910989474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:20:36.110425376 +0000 UTC m=+1731.500817992" watchObservedRunningTime="2026-07-23 22:20:36.111740423 +0000 UTC m=+1731.502133097"
Jul 23 22:20:36.812788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:36.812748    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:36.818048 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:36.818019    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:37.092897 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.092789    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:37.093945 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.093927    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:37.592433 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.592395    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:37.696169 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.696136    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"]
Jul 23 22:20:37.699492 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.699475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:37.701683 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.701658    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-58ac7283-serving-cert\""
Jul 23 22:20:37.713977 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.713934    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"]
Jul 23 22:20:37.803444 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.803413    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:37.806551 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.806526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmz9\" (UniqueName: \"kubernetes.io/projected/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-kube-api-access-qqmz9\") pod \"maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:37.806650 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.806571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-maas-api-tls\") pod \"maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:37.806650 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.806590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-openshift-service-ca\") pod \"maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:37.907305 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.907205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmz9\" (UniqueName: \"kubernetes.io/projected/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-kube-api-access-qqmz9\") pod \"maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:37.907305 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.907261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-maas-api-tls\") pod \"maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:37.907554 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:20:37.907351    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-serving-cert: secret "maas-api-e2e-migrate-58ac7283-serving-cert" not found
Jul 23 22:20:37.907554 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.907384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-openshift-service-ca\") pod \"maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:37.907554 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:20:37.907406    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-maas-api-tls podName:5dbbe79d-2201-4dbd-b7a2-ced883009a0f nodeName:}" failed. No retries permitted until 2026-07-23 22:20:38.407389201 +0000 UTC m=+1733.797781796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-maas-api-tls") pod "maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln" (UID: "5dbbe79d-2201-4dbd-b7a2-ced883009a0f") : secret "maas-api-e2e-migrate-58ac7283-serving-cert" not found
Jul 23 22:20:37.908128 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.908102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-openshift-service-ca\") pod \"maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:37.918227 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:37.918196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmz9\" (UniqueName: \"kubernetes.io/projected/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-kube-api-access-qqmz9\") pod \"maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:38.232706 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:38.232669    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:38.337614 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:38.337582    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:38.412121 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:38.412075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-maas-api-tls\") pod \"maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:38.414898 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:38.414853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-maas-api-tls\") pod \"maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:38.435541 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:38.435508    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:38.609643 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:38.609553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:38.753297 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:38.753272    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"]
Jul 23 22:20:38.755618 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:20:38.755594    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbbe79d_2201_4dbd_b7a2_ced883009a0f.slice/crio-280cda7d13966a1d15786c1b3edb7022b9c1f2344c70ba5955b554df0fe42f1b WatchSource:0}: Error finding container 280cda7d13966a1d15786c1b3edb7022b9c1f2344c70ba5955b554df0fe42f1b: Status 404 returned error can't find the container with id 280cda7d13966a1d15786c1b3edb7022b9c1f2344c70ba5955b554df0fe42f1b
Jul 23 22:20:39.103163 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:39.103122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln" event={"ID":"5dbbe79d-2201-4dbd-b7a2-ced883009a0f","Type":"ContainerStarted","Data":"280cda7d13966a1d15786c1b3edb7022b9c1f2344c70ba5955b554df0fe42f1b"}
Jul 23 22:20:40.109081 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:40.109049    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln" event={"ID":"5dbbe79d-2201-4dbd-b7a2-ced883009a0f","Type":"ContainerStarted","Data":"f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc"}
Jul 23 22:20:40.109482 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:40.109165    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:40.128027 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:40.127963    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln" podStartSLOduration=2.62179567 podStartE2EDuration="3.127948012s" totalImagesPullingTime="506.152342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:20:40.126109529 +0000 UTC m=+1735.516502146" watchObservedRunningTime="2026-07-23 22:20:40.127948012 +0000 UTC m=+1735.518340689"
Jul 23 22:20:46.118581 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:46.118552    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:20:49.147795 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:49.147714    2579 generic.go:356] "Generic (PLEG): container finished" podID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerID="ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d" exitCode=0
Jul 23 22:20:49.148208 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:49.147783    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" event={"ID":"e7e19d92-e199-4071-9dc8-c29d2abb2bdd","Type":"ContainerDied","Data":"ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d"}
Jul 23 22:20:49.148208 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:49.148186    2579 scope.go:117] "RemoveContainer" containerID="ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d"
Jul 23 22:20:49.150113 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:49.150085    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:20:49.150183 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:49.150164    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:20:49.150222 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:49.150195    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:20:50.154362 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:50.154325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" event={"ID":"e7e19d92-e199-4071-9dc8-c29d2abb2bdd","Type":"ContainerStarted","Data":"2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe"}
Jul 23 22:20:50.812781 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:50.812741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:50.813990 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:50.813954    2579 patch_prober.go:28] interesting pod/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 23 22:20:50.814124 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:50.814007    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 23 22:20:51.812751 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:51.812712    2579 patch_prober.go:28] interesting pod/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 23 22:20:51.813218 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:51.812784    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 23 22:20:52.093734 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:52.093571    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:52.812599 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:52.812557    2579 patch_prober.go:28] interesting pod/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 23 22:20:52.812788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:52.812623    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 23 22:20:53.394269 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:53.394216    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:53.454564 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:53.454525    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:20:53.813123 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:53.813080    2579 patch_prober.go:28] interesting pod/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 23 22:20:53.813514 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:53.813156    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 23 22:20:54.816700 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:54.816670    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:20:54.817940 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:20:54.817914    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:21:02.644389 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:02.644352    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"]
Jul 23 22:21:02.653808 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:02.653781    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"
Jul 23 22:21:02.656196 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:02.656167    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lq4mm\""
Jul 23 22:21:02.659564 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:02.658164    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"]
Jul 23 22:21:02.743232 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:02.743187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4717f0fa-b156-47c3-9abe-828dfc86778e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6\" (UID: \"4717f0fa-b156-47c3-9abe-828dfc86778e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"
Jul 23 22:21:02.844002 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:02.843958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4717f0fa-b156-47c3-9abe-828dfc86778e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6\" (UID: \"4717f0fa-b156-47c3-9abe-828dfc86778e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"
Jul 23 22:21:02.844752 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:02.844709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4717f0fa-b156-47c3-9abe-828dfc86778e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6\" (UID: \"4717f0fa-b156-47c3-9abe-828dfc86778e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"
Jul 23 22:21:02.971625 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:02.971584    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"
Jul 23 22:21:03.105550 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:03.105521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"]
Jul 23 22:21:03.108503 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:21:03.108473    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4717f0fa_b156_47c3_9abe_828dfc86778e.slice/crio-fc94c02158d856b44945962690eec05e0f7ab0c75c40a6310966618032cd9512 WatchSource:0}: Error finding container fc94c02158d856b44945962690eec05e0f7ab0c75c40a6310966618032cd9512: Status 404 returned error can't find the container with id fc94c02158d856b44945962690eec05e0f7ab0c75c40a6310966618032cd9512
Jul 23 22:21:03.211624 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:03.211583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6" event={"ID":"4717f0fa-b156-47c3-9abe-828dfc86778e","Type":"ContainerStarted","Data":"fc94c02158d856b44945962690eec05e0f7ab0c75c40a6310966618032cd9512"}
Jul 23 22:21:04.217714 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:04.217675    2579 generic.go:356] "Generic (PLEG): container finished" podID="4717f0fa-b156-47c3-9abe-828dfc86778e" containerID="d1aabb4d2c88d6a447a9755fa32ec6645e05cbdfd35b03cf7fb36e8acea11919" exitCode=0
Jul 23 22:21:04.218116 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:04.217761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6" event={"ID":"4717f0fa-b156-47c3-9abe-828dfc86778e","Type":"ContainerDied","Data":"d1aabb4d2c88d6a447a9755fa32ec6645e05cbdfd35b03cf7fb36e8acea11919"}
Jul 23 22:21:05.358383 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:05.358354    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"
Jul 23 22:21:05.466334 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:05.466307    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4717f0fa-b156-47c3-9abe-828dfc86778e-maas-api-service-ca\") pod \"4717f0fa-b156-47c3-9abe-828dfc86778e\" (UID: \"4717f0fa-b156-47c3-9abe-828dfc86778e\") "
Jul 23 22:21:05.466727 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:05.466697    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4717f0fa-b156-47c3-9abe-828dfc86778e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4717f0fa-b156-47c3-9abe-828dfc86778e" (UID: "4717f0fa-b156-47c3-9abe-828dfc86778e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:21:05.567486 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:05.567394    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4717f0fa-b156-47c3-9abe-828dfc86778e-maas-api-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:06.228350 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:06.228313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6" event={"ID":"4717f0fa-b156-47c3-9abe-828dfc86778e","Type":"ContainerDied","Data":"fc94c02158d856b44945962690eec05e0f7ab0c75c40a6310966618032cd9512"}
Jul 23 22:21:06.228350 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:06.228351    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"
Jul 23 22:21:06.228567 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:06.228364    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc94c02158d856b44945962690eec05e0f7ab0c75c40a6310966618032cd9512"
Jul 23 22:21:13.333988 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.333952    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"]
Jul 23 22:21:13.334383 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.334163    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln" podUID="5dbbe79d-2201-4dbd-b7a2-ced883009a0f" containerName="maas-api" containerID="cri-o://f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc" gracePeriod=30
Jul 23 22:21:13.361601 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.361554    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:13.581673 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.581650    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:21:13.636641 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.636554    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqmz9\" (UniqueName: \"kubernetes.io/projected/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-kube-api-access-qqmz9\") pod \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") "
Jul 23 22:21:13.636641 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.636644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-openshift-service-ca\") pod \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") "
Jul 23 22:21:13.636843 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.636664    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-maas-api-tls\") pod \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\" (UID: \"5dbbe79d-2201-4dbd-b7a2-ced883009a0f\") "
Jul 23 22:21:13.637067 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.637039    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5dbbe79d-2201-4dbd-b7a2-ced883009a0f" (UID: "5dbbe79d-2201-4dbd-b7a2-ced883009a0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:21:13.639031 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.639000    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5dbbe79d-2201-4dbd-b7a2-ced883009a0f" (UID: "5dbbe79d-2201-4dbd-b7a2-ced883009a0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:21:13.639031 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.639014    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-kube-api-access-qqmz9" (OuterVolumeSpecName: "kube-api-access-qqmz9") pod "5dbbe79d-2201-4dbd-b7a2-ced883009a0f" (UID: "5dbbe79d-2201-4dbd-b7a2-ced883009a0f"). InnerVolumeSpecName "kube-api-access-qqmz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:21:13.737625 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.737586    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmz9\" (UniqueName: \"kubernetes.io/projected/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-kube-api-access-qqmz9\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:13.737625 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.737618    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:13.737625 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:13.737627    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dbbe79d-2201-4dbd-b7a2-ced883009a0f-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:14.265952 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:14.265913    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dbbe79d-2201-4dbd-b7a2-ced883009a0f" containerID="f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc" exitCode=0
Jul 23 22:21:14.266147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:14.265988    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"
Jul 23 22:21:14.266147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:14.266011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln" event={"ID":"5dbbe79d-2201-4dbd-b7a2-ced883009a0f","Type":"ContainerDied","Data":"f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc"}
Jul 23 22:21:14.266147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:14.266097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln" event={"ID":"5dbbe79d-2201-4dbd-b7a2-ced883009a0f","Type":"ContainerDied","Data":"280cda7d13966a1d15786c1b3edb7022b9c1f2344c70ba5955b554df0fe42f1b"}
Jul 23 22:21:14.266147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:14.266119    2579 scope.go:117] "RemoveContainer" containerID="f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc"
Jul 23 22:21:14.276148 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:14.276120    2579 scope.go:117] "RemoveContainer" containerID="f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc"
Jul 23 22:21:14.276504 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:21:14.276483    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc\": container with ID starting with f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc not found: ID does not exist" containerID="f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc"
Jul 23 22:21:14.276589 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:14.276514    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc"} err="failed to get container status \"f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc\": rpc error: code = NotFound desc = could not find container \"f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc\": container with ID starting with f7c17babb0af9777ab031b11579da9951ce816661506559753635d957fd7a4bc not found: ID does not exist"
Jul 23 22:21:14.287880 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:14.287814    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"]
Jul 23 22:21:14.290998 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:14.290975    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-58ac7283-7548cbdbcd-ft2ln"]
Jul 23 22:21:15.115061 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:15.113221    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dbbe79d-2201-4dbd-b7a2-ced883009a0f" path="/var/lib/kubelet/pods/5dbbe79d-2201-4dbd-b7a2-ced883009a0f/volumes"
Jul 23 22:21:17.736809 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:17.736775    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"]
Jul 23 22:21:17.737424 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:17.737144    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:17.739924 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:17.739894    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-58ac7283-flhq6"]
Jul 23 22:21:17.869493 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:17.869460    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"]
Jul 23 22:21:17.869705 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:17.869677    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerName="istio-proxy" containerID="cri-o://2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe" gracePeriod=30
Jul 23 22:21:17.890597 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:17.890558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:18.128175 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:18.128082    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:18.373203 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:18.373172    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:18.714072 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:18.714038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:18.894578 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:18.894543    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:19.112278 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:19.112194    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4717f0fa-b156-47c3-9abe-828dfc86778e" path="/var/lib/kubelet/pods/4717f0fa-b156-47c3-9abe-828dfc86778e/volumes"
Jul 23 22:21:21.042914 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.042850    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:21.135455 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.135418    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"]
Jul 23 22:21:21.135821 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.135803    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbbe79d-2201-4dbd-b7a2-ced883009a0f" containerName="maas-api"
Jul 23 22:21:21.135941 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.135824    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbbe79d-2201-4dbd-b7a2-ced883009a0f" containerName="maas-api"
Jul 23 22:21:21.135941 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.135877    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4717f0fa-b156-47c3-9abe-828dfc86778e" containerName="revoke-keys"
Jul 23 22:21:21.135941 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.135886    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4717f0fa-b156-47c3-9abe-828dfc86778e" containerName="revoke-keys"
Jul 23 22:21:21.136103 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.135961    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbbe79d-2201-4dbd-b7a2-ced883009a0f" containerName="maas-api"
Jul 23 22:21:21.136103 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.135979    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4717f0fa-b156-47c3-9abe-828dfc86778e" containerName="revoke-keys"
Jul 23 22:21:21.138755 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.138732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:21.141043 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.141021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0cf92335-serving-cert\""
Jul 23 22:21:21.149035 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.148998    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"]
Jul 23 22:21:21.202438 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.202397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd37e229-4188-49fd-86ec-40ef72cedda2-openshift-service-ca\") pod \"maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:21.202621 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.202459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd37e229-4188-49fd-86ec-40ef72cedda2-maas-api-tls\") pod \"maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:21.202621 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.202500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzmcr\" (UniqueName: \"kubernetes.io/projected/fd37e229-4188-49fd-86ec-40ef72cedda2-kube-api-access-rzmcr\") pod \"maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:21.214631 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.214602    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:21.303266 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.303178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd37e229-4188-49fd-86ec-40ef72cedda2-maas-api-tls\") pod \"maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:21.303266 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.303223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzmcr\" (UniqueName: \"kubernetes.io/projected/fd37e229-4188-49fd-86ec-40ef72cedda2-kube-api-access-rzmcr\") pod \"maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:21.303464 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.303279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd37e229-4188-49fd-86ec-40ef72cedda2-openshift-service-ca\") pod \"maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:21.303464 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:21:21.303333    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-serving-cert: secret "maas-api-e2e-ait-0cf92335-serving-cert" not found
Jul 23 22:21:21.303464 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:21:21.303400    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd37e229-4188-49fd-86ec-40ef72cedda2-maas-api-tls podName:fd37e229-4188-49fd-86ec-40ef72cedda2 nodeName:}" failed. No retries permitted until 2026-07-23 22:21:21.803383809 +0000 UTC m=+1777.193776403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd37e229-4188-49fd-86ec-40ef72cedda2-maas-api-tls") pod "maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr" (UID: "fd37e229-4188-49fd-86ec-40ef72cedda2") : secret "maas-api-e2e-ait-0cf92335-serving-cert" not found
Jul 23 22:21:21.303884 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.303843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd37e229-4188-49fd-86ec-40ef72cedda2-openshift-service-ca\") pod \"maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:21.313468 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.313434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzmcr\" (UniqueName: \"kubernetes.io/projected/fd37e229-4188-49fd-86ec-40ef72cedda2-kube-api-access-rzmcr\") pod \"maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:21.322279 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.322250    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:21.574418 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.574339    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:21.712733 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.712693    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:21.808977 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.808934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd37e229-4188-49fd-86ec-40ef72cedda2-maas-api-tls\") pod \"maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:21.811658 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.811627    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd37e229-4188-49fd-86ec-40ef72cedda2-maas-api-tls\") pod \"maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:21.813068 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:21.813042    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:22.048982 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:22.048940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:22.190000 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:22.189920    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"]
Jul 23 22:21:22.193540 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:21:22.193503    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd37e229_4188_49fd_86ec_40ef72cedda2.slice/crio-89284176bbdf3669fb8defe633ce703a3d5dbeed2061c75fca508072db0db987 WatchSource:0}: Error finding container 89284176bbdf3669fb8defe633ce703a3d5dbeed2061c75fca508072db0db987: Status 404 returned error can't find the container with id 89284176bbdf3669fb8defe633ce703a3d5dbeed2061c75fca508072db0db987
Jul 23 22:21:22.309111 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:22.309026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr" event={"ID":"fd37e229-4188-49fd-86ec-40ef72cedda2","Type":"ContainerStarted","Data":"89284176bbdf3669fb8defe633ce703a3d5dbeed2061c75fca508072db0db987"}
Jul 23 22:21:23.118944 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.118917    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:21:23.222097 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222055    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-credential-socket\") pod \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") "
Jul 23 22:21:23.222296 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222107    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-envoy\") pod \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") "
Jul 23 22:21:23.222296 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222168    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-data\") pod \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") "
Jul 23 22:21:23.222296 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-socket\") pod \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") "
Jul 23 22:21:23.222296 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-certs\") pod \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") "
Jul 23 22:21:23.222296 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222245    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istiod-ca-cert\") pod \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") "
Jul 23 22:21:23.222296 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222271    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-token\") pod \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") "
Jul 23 22:21:23.222296 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222287    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-podinfo\") pod \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") "
Jul 23 22:21:23.222660 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsz8j\" (UniqueName: \"kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-kube-api-access-rsz8j\") pod \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\" (UID: \"e7e19d92-e199-4071-9dc8-c29d2abb2bdd\") "
Jul 23 22:21:23.222660 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222407    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e7e19d92-e199-4071-9dc8-c29d2abb2bdd" (UID: "e7e19d92-e199-4071-9dc8-c29d2abb2bdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:21:23.222660 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222499    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e7e19d92-e199-4071-9dc8-c29d2abb2bdd" (UID: "e7e19d92-e199-4071-9dc8-c29d2abb2bdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:21:23.222660 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222512    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e7e19d92-e199-4071-9dc8-c29d2abb2bdd" (UID: "e7e19d92-e199-4071-9dc8-c29d2abb2bdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:21:23.222901 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222665    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:23.222901 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222683    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-workload-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:23.222901 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222696    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-credential-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:23.223149 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.222974    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e7e19d92-e199-4071-9dc8-c29d2abb2bdd" (UID: "e7e19d92-e199-4071-9dc8-c29d2abb2bdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:21:23.223286 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.223199    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-data" (OuterVolumeSpecName: "istio-data") pod "e7e19d92-e199-4071-9dc8-c29d2abb2bdd" (UID: "e7e19d92-e199-4071-9dc8-c29d2abb2bdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:21:23.224782 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.224745    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e7e19d92-e199-4071-9dc8-c29d2abb2bdd" (UID: "e7e19d92-e199-4071-9dc8-c29d2abb2bdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:21:23.224960 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.224803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e7e19d92-e199-4071-9dc8-c29d2abb2bdd" (UID: "e7e19d92-e199-4071-9dc8-c29d2abb2bdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 22:21:23.224960 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.224827    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-kube-api-access-rsz8j" (OuterVolumeSpecName: "kube-api-access-rsz8j") pod "e7e19d92-e199-4071-9dc8-c29d2abb2bdd" (UID: "e7e19d92-e199-4071-9dc8-c29d2abb2bdd"). InnerVolumeSpecName "kube-api-access-rsz8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:21:23.224960 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.224838    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-token" (OuterVolumeSpecName: "istio-token") pod "e7e19d92-e199-4071-9dc8-c29d2abb2bdd" (UID: "e7e19d92-e199-4071-9dc8-c29d2abb2bdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:21:23.315797 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.315748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr" event={"ID":"fd37e229-4188-49fd-86ec-40ef72cedda2","Type":"ContainerStarted","Data":"dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550"}
Jul 23 22:21:23.316007 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.315827    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:23.317425 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.317401    2579 generic.go:356] "Generic (PLEG): container finished" podID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerID="2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe" exitCode=0
Jul 23 22:21:23.317516 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.317450    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"
Jul 23 22:21:23.317569 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.317453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" event={"ID":"e7e19d92-e199-4071-9dc8-c29d2abb2bdd","Type":"ContainerDied","Data":"2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe"}
Jul 23 22:21:23.317605 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.317574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls" event={"ID":"e7e19d92-e199-4071-9dc8-c29d2abb2bdd","Type":"ContainerDied","Data":"111e15e6e00904eb9fee1599696c8f30b5eb1cc89420e6e34ad80b64534f9a6c"}
Jul 23 22:21:23.317605 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.317602    2579 scope.go:117] "RemoveContainer" containerID="2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe"
Jul 23 22:21:23.323456 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.323432    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-data\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:23.323569 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.323458    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istiod-ca-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:23.323569 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.323470    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-token\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:23.323569 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.323482    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-podinfo\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:23.323569 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.323494    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsz8j\" (UniqueName: \"kubernetes.io/projected/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-kube-api-access-rsz8j\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:23.323569 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.323505    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e19d92-e199-4071-9dc8-c29d2abb2bdd-istio-envoy\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:23.326940 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.326921    2579 scope.go:117] "RemoveContainer" containerID="ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d"
Jul 23 22:21:23.335314 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.335255    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr" podStartSLOduration=1.906599009 podStartE2EDuration="2.33523789s" totalImagesPullingTime="428.638881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:21:23.333448437 +0000 UTC m=+1778.723841066" watchObservedRunningTime="2026-07-23 22:21:23.33523789 +0000 UTC m=+1778.725630506"
Jul 23 22:21:23.336875 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.336843    2579 scope.go:117] "RemoveContainer" containerID="2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe"
Jul 23 22:21:23.337205 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:21:23.337185    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe\": container with ID starting with 2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe not found: ID does not exist" containerID="2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe"
Jul 23 22:21:23.337258 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.337221    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe"} err="failed to get container status \"2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe\": rpc error: code = NotFound desc = could not find container \"2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe\": container with ID starting with 2680e1881a819a88c857f72f34e3642e9f0f7a4c3edf7de03392ba50f7f628fe not found: ID does not exist"
Jul 23 22:21:23.337258 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.337240    2579 scope.go:117] "RemoveContainer" containerID="ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d"
Jul 23 22:21:23.337503 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:21:23.337487    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d\": container with ID starting with ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d not found: ID does not exist" containerID="ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d"
Jul 23 22:21:23.337558 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.337505    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d"} err="failed to get container status \"ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d\": rpc error: code = NotFound desc = could not find container \"ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d\": container with ID starting with ef65055fd935178d47e59e1b4bf9dc01954b7ebcf233b229617c3ae64cab2e4d not found: ID does not exist"
Jul 23 22:21:23.349637 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.349597    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"]
Jul 23 22:21:23.355915 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:23.355881    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-58ac7283-gw-openshift-default-66d6d64587-bx8ls"]
Jul 23 22:21:25.112179 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:25.112142    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" path="/var/lib/kubelet/pods/e7e19d92-e199-4071-9dc8-c29d2abb2bdd/volumes"
Jul 23 22:21:29.329341 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:29.329308    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:36.907716 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:36.907634    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:37.052003 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:37.051968    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:44.725877 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:44.725838    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:44.826809 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:44.826772    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:45.106935 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:45.106828    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:21:45.107449 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:45.107425    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:21:46.927914 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:46.927856    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"]
Jul 23 22:21:46.928308 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:46.928226    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerName="istio-proxy"
Jul 23 22:21:46.928308 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:46.928240    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerName="istio-proxy"
Jul 23 22:21:46.928308 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:46.928261    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerName="istio-proxy"
Jul 23 22:21:46.928308 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:46.928268    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerName="istio-proxy"
Jul 23 22:21:46.928454 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:46.928313    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerName="istio-proxy"
Jul 23 22:21:46.928454 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:46.928325    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e19d92-e199-4071-9dc8-c29d2abb2bdd" containerName="istio-proxy"
Jul 23 22:21:46.932507 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:46.932490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"
Jul 23 22:21:46.938825 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:46.938741    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lq4mm\""
Jul 23 22:21:46.949528 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:46.949492    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"]
Jul 23 22:21:47.028178 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:47.028134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c3d6a3-f9c3-4e86-860b-35dec762f4cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0cf92335-2gk72\" (UID: \"47c3d6a3-f9c3-4e86-860b-35dec762f4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"
Jul 23 22:21:47.129082 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:47.129041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c3d6a3-f9c3-4e86-860b-35dec762f4cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0cf92335-2gk72\" (UID: \"47c3d6a3-f9c3-4e86-860b-35dec762f4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"
Jul 23 22:21:47.129668 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:47.129647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c3d6a3-f9c3-4e86-860b-35dec762f4cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0cf92335-2gk72\" (UID: \"47c3d6a3-f9c3-4e86-860b-35dec762f4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"
Jul 23 22:21:47.242317 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:47.242276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"
Jul 23 22:21:47.377407 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:47.377377    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"]
Jul 23 22:21:47.379266 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:21:47.379230    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c3d6a3_f9c3_4e86_860b_35dec762f4cb.slice/crio-3ab97d3155a0e25c87c033bb05534ef1eff50ada43a11f2d3f488e34cbde568b WatchSource:0}: Error finding container 3ab97d3155a0e25c87c033bb05534ef1eff50ada43a11f2d3f488e34cbde568b: Status 404 returned error can't find the container with id 3ab97d3155a0e25c87c033bb05534ef1eff50ada43a11f2d3f488e34cbde568b
Jul 23 22:21:47.419430 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:47.419398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72" event={"ID":"47c3d6a3-f9c3-4e86-860b-35dec762f4cb","Type":"ContainerStarted","Data":"3ab97d3155a0e25c87c033bb05534ef1eff50ada43a11f2d3f488e34cbde568b"}
Jul 23 22:21:48.425463 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:48.425359    2579 generic.go:356] "Generic (PLEG): container finished" podID="47c3d6a3-f9c3-4e86-860b-35dec762f4cb" containerID="3adb8ad25c536e4101e403ccd26ee8cf9245926018b0c8a2b46ce1805b5e57de" exitCode=0
Jul 23 22:21:48.425463 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:48.425452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72" event={"ID":"47c3d6a3-f9c3-4e86-860b-35dec762f4cb","Type":"ContainerDied","Data":"3adb8ad25c536e4101e403ccd26ee8cf9245926018b0c8a2b46ce1805b5e57de"}
Jul 23 22:21:49.551404 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:49.551377    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"
Jul 23 22:21:49.651368 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:49.651325    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c3d6a3-f9c3-4e86-860b-35dec762f4cb-maas-api-service-ca\") pod \"47c3d6a3-f9c3-4e86-860b-35dec762f4cb\" (UID: \"47c3d6a3-f9c3-4e86-860b-35dec762f4cb\") "
Jul 23 22:21:49.651745 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:49.651714    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c3d6a3-f9c3-4e86-860b-35dec762f4cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47c3d6a3-f9c3-4e86-860b-35dec762f4cb" (UID: "47c3d6a3-f9c3-4e86-860b-35dec762f4cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:21:49.752302 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:49.752251    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47c3d6a3-f9c3-4e86-860b-35dec762f4cb-maas-api-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:50.435927 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:50.435887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72" event={"ID":"47c3d6a3-f9c3-4e86-860b-35dec762f4cb","Type":"ContainerDied","Data":"3ab97d3155a0e25c87c033bb05534ef1eff50ada43a11f2d3f488e34cbde568b"}
Jul 23 22:21:50.435927 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:50.435913    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"
Jul 23 22:21:50.436151 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:50.435939    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ab97d3155a0e25c87c033bb05534ef1eff50ada43a11f2d3f488e34cbde568b"
Jul 23 22:21:57.460843 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.460807    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"]
Jul 23 22:21:57.461384 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.461021    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr" podUID="fd37e229-4188-49fd-86ec-40ef72cedda2" containerName="maas-api" containerID="cri-o://dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550" gracePeriod=30
Jul 23 22:21:57.503566 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.503513    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:21:57.713970 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.713903    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:57.821323 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.821285    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd37e229-4188-49fd-86ec-40ef72cedda2-maas-api-tls\") pod \"fd37e229-4188-49fd-86ec-40ef72cedda2\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") "
Jul 23 22:21:57.821493 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.821342    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd37e229-4188-49fd-86ec-40ef72cedda2-openshift-service-ca\") pod \"fd37e229-4188-49fd-86ec-40ef72cedda2\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") "
Jul 23 22:21:57.821493 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.821391    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzmcr\" (UniqueName: \"kubernetes.io/projected/fd37e229-4188-49fd-86ec-40ef72cedda2-kube-api-access-rzmcr\") pod \"fd37e229-4188-49fd-86ec-40ef72cedda2\" (UID: \"fd37e229-4188-49fd-86ec-40ef72cedda2\") "
Jul 23 22:21:57.821768 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.821730    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd37e229-4188-49fd-86ec-40ef72cedda2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd37e229-4188-49fd-86ec-40ef72cedda2" (UID: "fd37e229-4188-49fd-86ec-40ef72cedda2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:21:57.823532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.823503    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd37e229-4188-49fd-86ec-40ef72cedda2-kube-api-access-rzmcr" (OuterVolumeSpecName: "kube-api-access-rzmcr") pod "fd37e229-4188-49fd-86ec-40ef72cedda2" (UID: "fd37e229-4188-49fd-86ec-40ef72cedda2"). InnerVolumeSpecName "kube-api-access-rzmcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:21:57.823610 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.823555    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd37e229-4188-49fd-86ec-40ef72cedda2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd37e229-4188-49fd-86ec-40ef72cedda2" (UID: "fd37e229-4188-49fd-86ec-40ef72cedda2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:21:57.922367 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.922323    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd37e229-4188-49fd-86ec-40ef72cedda2-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:57.922367 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.922361    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd37e229-4188-49fd-86ec-40ef72cedda2-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:57.922367 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:57.922375    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzmcr\" (UniqueName: \"kubernetes.io/projected/fd37e229-4188-49fd-86ec-40ef72cedda2-kube-api-access-rzmcr\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:21:58.470534 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:58.470494    2579 generic.go:356] "Generic (PLEG): container finished" podID="fd37e229-4188-49fd-86ec-40ef72cedda2" containerID="dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550" exitCode=0
Jul 23 22:21:58.470955 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:58.470540    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"
Jul 23 22:21:58.470955 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:58.470581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr" event={"ID":"fd37e229-4188-49fd-86ec-40ef72cedda2","Type":"ContainerDied","Data":"dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550"}
Jul 23 22:21:58.470955 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:58.470639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr" event={"ID":"fd37e229-4188-49fd-86ec-40ef72cedda2","Type":"ContainerDied","Data":"89284176bbdf3669fb8defe633ce703a3d5dbeed2061c75fca508072db0db987"}
Jul 23 22:21:58.470955 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:58.470662    2579 scope.go:117] "RemoveContainer" containerID="dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550"
Jul 23 22:21:58.479605 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:58.479588    2579 scope.go:117] "RemoveContainer" containerID="dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550"
Jul 23 22:21:58.479928 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:21:58.479905    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550\": container with ID starting with dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550 not found: ID does not exist" containerID="dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550"
Jul 23 22:21:58.480018 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:58.479936    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550"} err="failed to get container status \"dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550\": rpc error: code = NotFound desc = could not find container \"dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550\": container with ID starting with dff87d7437e0489351395a2e70ba2a394dec4dbe529c63ebb149ea861cf80550 not found: ID does not exist"
Jul 23 22:21:58.491117 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:58.491092    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"]
Jul 23 22:21:58.494842 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:58.494820    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0cf92335-b54f998cf-6qrcr"]
Jul 23 22:21:59.111460 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:21:59.111424    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd37e229-4188-49fd-86ec-40ef72cedda2" path="/var/lib/kubelet/pods/fd37e229-4188-49fd-86ec-40ef72cedda2/volumes"
Jul 23 22:22:02.009351 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:02.009319    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"]
Jul 23 22:22:02.009757 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:02.009720    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:02.013946 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:02.013923    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0cf92335-2gk72"]
Jul 23 22:22:03.110983 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:03.110951    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47c3d6a3-f9c3-4e86-860b-35dec762f4cb" path="/var/lib/kubelet/pods/47c3d6a3-f9c3-4e86-860b-35dec762f4cb/volumes"
Jul 23 22:22:03.307459 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:03.307428    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:03.540851 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:03.540816    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:03.785356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:03.785325    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:04.129093 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.129056    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"]
Jul 23 22:22:04.129479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.129346    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47c3d6a3-f9c3-4e86-860b-35dec762f4cb" containerName="revoke-keys"
Jul 23 22:22:04.129479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.129358    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="47c3d6a3-f9c3-4e86-860b-35dec762f4cb" containerName="revoke-keys"
Jul 23 22:22:04.129479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.129382    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd37e229-4188-49fd-86ec-40ef72cedda2" containerName="maas-api"
Jul 23 22:22:04.129479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.129389    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd37e229-4188-49fd-86ec-40ef72cedda2" containerName="maas-api"
Jul 23 22:22:04.129479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.129433    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd37e229-4188-49fd-86ec-40ef72cedda2" containerName="maas-api"
Jul 23 22:22:04.129479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.129443    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="47c3d6a3-f9c3-4e86-860b-35dec762f4cb" containerName="revoke-keys"
Jul 23 22:22:04.133880 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.133843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.136178 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.136146    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 22:22:04.136318 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.136279    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1b6e6ae2-openshift-default-dockercfg-4h9b5\""
Jul 23 22:22:04.148486 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.148447    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"]
Jul 23 22:22:04.165503 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.165461    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:04.282286 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.282245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-token\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.282286 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.282278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-envoy\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.282286 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.282298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-podinfo\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.282610 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.282383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-socket\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.282610 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.282417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b7e9a7-5260-4dda-aba4-87908f97e48b-istiod-ca-cert\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.282610 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.282436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-data\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.282610 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.282502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-certs\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.282610 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.282529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-credential-socket\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.282610 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.282558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98qk\" (UniqueName: \"kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-kube-api-access-g98qk\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383325 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383230    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b7e9a7-5260-4dda-aba4-87908f97e48b-istiod-ca-cert\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383325 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-data\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383325 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-certs\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383584 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-credential-socket\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383584 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98qk\" (UniqueName: \"kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-kube-api-access-g98qk\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383584 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-token\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383737 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-envoy\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383737 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-podinfo\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383737 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-socket\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383923 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-credential-socket\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383923 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-data\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.383923 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.383906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-certs\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.384155 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.384129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b7e9a7-5260-4dda-aba4-87908f97e48b-istiod-ca-cert\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.384271 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.384131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-socket\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.386144 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.386124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-envoy\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.386281 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.386262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-podinfo\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.393929 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.393898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-token\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.394097 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.394079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98qk\" (UniqueName: \"kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-kube-api-access-g98qk\") pod \"e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") " pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.443928 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.443893    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:04.555374 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.555342    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:04.591095 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.591067    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"]
Jul 23 22:22:04.593260 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:22:04.593230    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b7e9a7_5260_4dda_aba4_87908f97e48b.slice/crio-f1891d3c5234acc206aaf074d88d8a3ccfdbdbded3ef558a6317f3d41977f946 WatchSource:0}: Error finding container f1891d3c5234acc206aaf074d88d8a3ccfdbdbded3ef558a6317f3d41977f946: Status 404 returned error can't find the container with id f1891d3c5234acc206aaf074d88d8a3ccfdbdbded3ef558a6317f3d41977f946
Jul 23 22:22:04.595292 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.595252    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:22:04.595371 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.595333    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:22:04.595410 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:04.595367    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:22:05.504128 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:05.504089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" event={"ID":"48b7e9a7-5260-4dda-aba4-87908f97e48b","Type":"ContainerStarted","Data":"e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7"}
Jul 23 22:22:05.504526 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:05.504148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" event={"ID":"48b7e9a7-5260-4dda-aba4-87908f97e48b","Type":"ContainerStarted","Data":"f1891d3c5234acc206aaf074d88d8a3ccfdbdbded3ef558a6317f3d41977f946"}
Jul 23 22:22:05.526830 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:05.526769    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" podStartSLOduration=1.526753054 podStartE2EDuration="1.526753054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:05.52573489 +0000 UTC m=+1820.916127508" watchObservedRunningTime="2026-07-23 22:22:05.526753054 +0000 UTC m=+1820.917145669"
Jul 23 22:22:06.444281 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:06.444243    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:06.449151 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:06.449115    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:06.508948 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:06.508916    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:06.509902 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:06.509879    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:06.979520 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:06.979493    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:07.070697 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.070666    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"]
Jul 23 22:22:07.074087 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.074064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:07.077544 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.077522    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-1b6e6ae2-serving-cert\""
Jul 23 22:22:07.087911 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.087852    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"]
Jul 23 22:22:07.115630 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.115603    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:07.210162 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.210125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-openshift-service-ca\") pod \"maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:07.210356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.210181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls\") pod \"maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:07.210356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.210248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b2zh\" (UniqueName: \"kubernetes.io/projected/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-kube-api-access-2b2zh\") pod \"maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:07.218781 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.218750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:07.310942 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.310826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls\") pod \"maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:07.310942 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.310898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b2zh\" (UniqueName: \"kubernetes.io/projected/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-kube-api-access-2b2zh\") pod \"maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:07.311147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.310959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-openshift-service-ca\") pod \"maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:07.311147 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:22:07.310983    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-serving-cert: secret "maas-api-e2e-derive-1b6e6ae2-serving-cert" not found
Jul 23 22:22:07.311147 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:22:07.311058    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls podName:b9f5726c-07f9-48ed-8ab6-632d52bba9e9 nodeName:}" failed. No retries permitted until 2026-07-23 22:22:07.811037461 +0000 UTC m=+1823.201430054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls") pod "maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn" (UID: "b9f5726c-07f9-48ed-8ab6-632d52bba9e9") : secret "maas-api-e2e-derive-1b6e6ae2-serving-cert" not found
Jul 23 22:22:07.311550 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.311532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-openshift-service-ca\") pod \"maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:07.322964 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.322932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b2zh\" (UniqueName: \"kubernetes.io/projected/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-kube-api-access-2b2zh\") pod \"maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:07.664742 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.664657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:07.764978 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.764931    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:07.815108 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:07.815057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls\") pod \"maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:07.815320 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:22:07.815220    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-serving-cert: secret "maas-api-e2e-derive-1b6e6ae2-serving-cert" not found
Jul 23 22:22:07.815320 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:22:07.815292    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls podName:b9f5726c-07f9-48ed-8ab6-632d52bba9e9 nodeName:}" failed. No retries permitted until 2026-07-23 22:22:08.815274923 +0000 UTC m=+1824.205667519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls") pod "maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn" (UID: "b9f5726c-07f9-48ed-8ab6-632d52bba9e9") : secret "maas-api-e2e-derive-1b6e6ae2-serving-cert" not found
Jul 23 22:22:08.385135 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:08.385099    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:08.824540 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:08.824496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls\") pod \"maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:08.827085 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:08.827060    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls\") pod \"maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:08.885362 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:08.885320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:09.025257 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:09.025218    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"]
Jul 23 22:22:09.029633 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:22:09.029599    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f5726c_07f9_48ed_8ab6_632d52bba9e9.slice/crio-9d5834fb1956b07416c72b5ccfd41eb78597df12de57b49fc808c6d572da536f WatchSource:0}: Error finding container 9d5834fb1956b07416c72b5ccfd41eb78597df12de57b49fc808c6d572da536f: Status 404 returned error can't find the container with id 9d5834fb1956b07416c72b5ccfd41eb78597df12de57b49fc808c6d572da536f
Jul 23 22:22:09.523058 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:09.523023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn" event={"ID":"b9f5726c-07f9-48ed-8ab6-632d52bba9e9","Type":"ContainerStarted","Data":"9d5834fb1956b07416c72b5ccfd41eb78597df12de57b49fc808c6d572da536f"}
Jul 23 22:22:10.529320 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:10.529282    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn" event={"ID":"b9f5726c-07f9-48ed-8ab6-632d52bba9e9","Type":"ContainerStarted","Data":"588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1"}
Jul 23 22:22:10.529800 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:10.529368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:10.546649 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:10.546592    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn" podStartSLOduration=3.073686384 podStartE2EDuration="3.54657715s" totalImagesPullingTime="472.890766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:10.545589008 +0000 UTC m=+1825.935981624" watchObservedRunningTime="2026-07-23 22:22:10.54657715 +0000 UTC m=+1825.936969773"
Jul 23 22:22:16.539474 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:16.539441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:17.532780 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:17.532748    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:21.510200 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:21.510106    2579 patch_prober.go:28] interesting pod/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 23 22:22:21.510200 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:21.510182    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 23 22:22:21.581201 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:21.581164    2579 generic.go:356] "Generic (PLEG): container finished" podID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerID="e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7" exitCode=0
Jul 23 22:22:21.581373 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:21.581212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" event={"ID":"48b7e9a7-5260-4dda-aba4-87908f97e48b","Type":"ContainerDied","Data":"e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7"}
Jul 23 22:22:21.581706 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:21.581686    2579 scope.go:117] "RemoveContainer" containerID="e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7"
Jul 23 22:22:21.582942 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:21.582908    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:22:21.583028 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:21.582999    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:22:21.583083 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:21.583058    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:22:22.591007 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:22.590956    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" event={"ID":"48b7e9a7-5260-4dda-aba4-87908f97e48b","Type":"ContainerStarted","Data":"d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088"}
Jul 23 22:22:23.444356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:23.444309    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:23.444664 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:23.444546    2579 patch_prober.go:28] interesting pod/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 23 22:22:23.444664 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:23.444611    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 23 22:22:24.444819 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:24.444782    2579 patch_prober.go:28] interesting pod/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 23 22:22:24.445242 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:24.444842    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 23 22:22:25.445355 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:25.445317    2579 patch_prober.go:28] interesting pod/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 23 22:22:25.445827 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:25.445383    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 23 22:22:26.449443 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:26.449412    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:26.449808 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:26.449749    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:26.450468 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:26.450450    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:31.846814 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:31.846780    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"]
Jul 23 22:22:31.850171 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:31.850152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"
Jul 23 22:22:31.852328 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:31.852303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lq4mm\""
Jul 23 22:22:31.858764 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:31.858723    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"]
Jul 23 22:22:31.934298 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:31.934248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4dca755-155f-426d-ad71-fc2e2eb867a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7\" (UID: \"d4dca755-155f-426d-ad71-fc2e2eb867a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"
Jul 23 22:22:32.035205 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:32.035168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4dca755-155f-426d-ad71-fc2e2eb867a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7\" (UID: \"d4dca755-155f-426d-ad71-fc2e2eb867a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"
Jul 23 22:22:32.035821 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:32.035800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4dca755-155f-426d-ad71-fc2e2eb867a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7\" (UID: \"d4dca755-155f-426d-ad71-fc2e2eb867a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"
Jul 23 22:22:32.162225 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:32.162133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"
Jul 23 22:22:32.323243 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:32.323216    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"]
Jul 23 22:22:32.325889 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:22:32.325845    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4dca755_155f_426d_ad71_fc2e2eb867a8.slice/crio-15e96c28264b2239163b08ac3b56e21ef2ecb45220f75b6995478a3e231bcaea WatchSource:0}: Error finding container 15e96c28264b2239163b08ac3b56e21ef2ecb45220f75b6995478a3e231bcaea: Status 404 returned error can't find the container with id 15e96c28264b2239163b08ac3b56e21ef2ecb45220f75b6995478a3e231bcaea
Jul 23 22:22:32.636975 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:32.636938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7" event={"ID":"d4dca755-155f-426d-ad71-fc2e2eb867a8","Type":"ContainerStarted","Data":"1f6922666c5e28f8e3e3fedebf2daae66af4067e2b4e2a83afc653e4cc8b8b2e"}
Jul 23 22:22:32.637151 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:32.637005    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7" event={"ID":"d4dca755-155f-426d-ad71-fc2e2eb867a8","Type":"ContainerStarted","Data":"15e96c28264b2239163b08ac3b56e21ef2ecb45220f75b6995478a3e231bcaea"}
Jul 23 22:22:32.651875 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:32.651806    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7" podStartSLOduration=1.6517903280000001 podStartE2EDuration="1.651790328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:22:32.650388056 +0000 UTC m=+1848.040780672" watchObservedRunningTime="2026-07-23 22:22:32.651790328 +0000 UTC m=+1848.042182944"
Jul 23 22:22:33.643033 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:33.642994    2579 generic.go:356] "Generic (PLEG): container finished" podID="d4dca755-155f-426d-ad71-fc2e2eb867a8" containerID="1f6922666c5e28f8e3e3fedebf2daae66af4067e2b4e2a83afc653e4cc8b8b2e" exitCode=0
Jul 23 22:22:33.643488 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:33.643084    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7" event={"ID":"d4dca755-155f-426d-ad71-fc2e2eb867a8","Type":"ContainerDied","Data":"1f6922666c5e28f8e3e3fedebf2daae66af4067e2b4e2a83afc653e4cc8b8b2e"}
Jul 23 22:22:34.777783 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:34.777758    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"
Jul 23 22:22:34.855641 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:34.855605    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4dca755-155f-426d-ad71-fc2e2eb867a8-maas-api-service-ca\") pod \"d4dca755-155f-426d-ad71-fc2e2eb867a8\" (UID: \"d4dca755-155f-426d-ad71-fc2e2eb867a8\") "
Jul 23 22:22:34.855984 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:34.855960    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4dca755-155f-426d-ad71-fc2e2eb867a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4dca755-155f-426d-ad71-fc2e2eb867a8" (UID: "d4dca755-155f-426d-ad71-fc2e2eb867a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:22:34.956960 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:34.956915    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4dca755-155f-426d-ad71-fc2e2eb867a8-maas-api-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:35.653136 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:35.653107    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"
Jul 23 22:22:35.653318 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:35.653105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7" event={"ID":"d4dca755-155f-426d-ad71-fc2e2eb867a8","Type":"ContainerDied","Data":"15e96c28264b2239163b08ac3b56e21ef2ecb45220f75b6995478a3e231bcaea"}
Jul 23 22:22:35.653318 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:35.653236    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15e96c28264b2239163b08ac3b56e21ef2ecb45220f75b6995478a3e231bcaea"
Jul 23 22:22:42.427639 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.427600    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"]
Jul 23 22:22:42.428040 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.427809    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn" podUID="b9f5726c-07f9-48ed-8ab6-632d52bba9e9" containerName="maas-api" containerID="cri-o://588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1" gracePeriod=30
Jul 23 22:22:42.449160 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.449129    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:42.676602 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.676573    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:42.687045 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.687005    2579 generic.go:356] "Generic (PLEG): container finished" podID="b9f5726c-07f9-48ed-8ab6-632d52bba9e9" containerID="588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1" exitCode=0
Jul 23 22:22:42.687210 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.687053    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"
Jul 23 22:22:42.687210 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.687086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn" event={"ID":"b9f5726c-07f9-48ed-8ab6-632d52bba9e9","Type":"ContainerDied","Data":"588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1"}
Jul 23 22:22:42.687210 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.687143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn" event={"ID":"b9f5726c-07f9-48ed-8ab6-632d52bba9e9","Type":"ContainerDied","Data":"9d5834fb1956b07416c72b5ccfd41eb78597df12de57b49fc808c6d572da536f"}
Jul 23 22:22:42.687210 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.687165    2579 scope.go:117] "RemoveContainer" containerID="588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1"
Jul 23 22:22:42.695582 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.695555    2579 scope.go:117] "RemoveContainer" containerID="588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1"
Jul 23 22:22:42.696011 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:22:42.695984    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1\": container with ID starting with 588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1 not found: ID does not exist" containerID="588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1"
Jul 23 22:22:42.696112 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.696025    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1"} err="failed to get container status \"588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1\": rpc error: code = NotFound desc = could not find container \"588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1\": container with ID starting with 588290410b7983b74f9b6491351d30eaa618c612c260bef55a3567f96f7dfba1 not found: ID does not exist"
Jul 23 22:22:42.717489 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.717453    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b2zh\" (UniqueName: \"kubernetes.io/projected/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-kube-api-access-2b2zh\") pod \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") "
Jul 23 22:22:42.717679 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.717512    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-openshift-service-ca\") pod \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") "
Jul 23 22:22:42.717679 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.717532    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls\") pod \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\" (UID: \"b9f5726c-07f9-48ed-8ab6-632d52bba9e9\") "
Jul 23 22:22:42.717998 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.717968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9f5726c-07f9-48ed-8ab6-632d52bba9e9" (UID: "b9f5726c-07f9-48ed-8ab6-632d52bba9e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:22:42.719797 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.719766    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9f5726c-07f9-48ed-8ab6-632d52bba9e9" (UID: "b9f5726c-07f9-48ed-8ab6-632d52bba9e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:22:42.719926 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.719892    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-kube-api-access-2b2zh" (OuterVolumeSpecName: "kube-api-access-2b2zh") pod "b9f5726c-07f9-48ed-8ab6-632d52bba9e9" (UID: "b9f5726c-07f9-48ed-8ab6-632d52bba9e9"). InnerVolumeSpecName "kube-api-access-2b2zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:22:42.818241 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.818201    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b2zh\" (UniqueName: \"kubernetes.io/projected/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-kube-api-access-2b2zh\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:42.818241 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.818233    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:42.818241 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:42.818244    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f5726c-07f9-48ed-8ab6-632d52bba9e9-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:43.007822 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:43.007790    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"]
Jul 23 22:22:43.014526 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:43.014486    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-1b6e6ae2-56476fff57-958qn"]
Jul 23 22:22:43.111732 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:43.111697    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9f5726c-07f9-48ed-8ab6-632d52bba9e9" path="/var/lib/kubelet/pods/b9f5726c-07f9-48ed-8ab6-632d52bba9e9/volumes"
Jul 23 22:22:46.951992 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:46.951952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:46.952463 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:46.952443    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"]
Jul 23 22:22:46.957301 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:46.957278    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b6e6ae2-7mcj7"]
Jul 23 22:22:47.113085 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:47.113053    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4dca755-155f-426d-ad71-fc2e2eb867a8" path="/var/lib/kubelet/pods/d4dca755-155f-426d-ad71-fc2e2eb867a8/volumes"
Jul 23 22:22:47.131511 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:47.131477    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"]
Jul 23 22:22:47.131730 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:47.131707    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerName="istio-proxy" containerID="cri-o://d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088" gracePeriod=30
Jul 23 22:22:47.143252 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:47.143223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:47.370271 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:47.370177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:47.901684 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:47.901642    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:48.000500 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:48.000451    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:48.102193 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:48.102152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:49.603113 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:49.603071    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:52.388950 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.388922    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:52.503888 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.503830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-token\") pod \"48b7e9a7-5260-4dda-aba4-87908f97e48b\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") "
Jul 23 22:22:52.504099 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.503925    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-credential-socket\") pod \"48b7e9a7-5260-4dda-aba4-87908f97e48b\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") "
Jul 23 22:22:52.504099 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.503961    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-certs\") pod \"48b7e9a7-5260-4dda-aba4-87908f97e48b\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") "
Jul 23 22:22:52.504099 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504008    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-podinfo\") pod \"48b7e9a7-5260-4dda-aba4-87908f97e48b\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") "
Jul 23 22:22:52.504099 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504037    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b7e9a7-5260-4dda-aba4-87908f97e48b-istiod-ca-cert\") pod \"48b7e9a7-5260-4dda-aba4-87908f97e48b\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") "
Jul 23 22:22:52.504099 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504096    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-envoy\") pod \"48b7e9a7-5260-4dda-aba4-87908f97e48b\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") "
Jul 23 22:22:52.504357 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504119    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g98qk\" (UniqueName: \"kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-kube-api-access-g98qk\") pod \"48b7e9a7-5260-4dda-aba4-87908f97e48b\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") "
Jul 23 22:22:52.504357 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504148    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-socket\") pod \"48b7e9a7-5260-4dda-aba4-87908f97e48b\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") "
Jul 23 22:22:52.504357 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504175    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-data\") pod \"48b7e9a7-5260-4dda-aba4-87908f97e48b\" (UID: \"48b7e9a7-5260-4dda-aba4-87908f97e48b\") "
Jul 23 22:22:52.504357 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504239    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "48b7e9a7-5260-4dda-aba4-87908f97e48b" (UID: "48b7e9a7-5260-4dda-aba4-87908f97e48b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:22:52.504564 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504358    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "48b7e9a7-5260-4dda-aba4-87908f97e48b" (UID: "48b7e9a7-5260-4dda-aba4-87908f97e48b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:22:52.504564 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504459    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b7e9a7-5260-4dda-aba4-87908f97e48b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "48b7e9a7-5260-4dda-aba4-87908f97e48b" (UID: "48b7e9a7-5260-4dda-aba4-87908f97e48b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:22:52.504564 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504487    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-credential-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:52.504564 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504507    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:52.504564 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.504524    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "48b7e9a7-5260-4dda-aba4-87908f97e48b" (UID: "48b7e9a7-5260-4dda-aba4-87908f97e48b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:22:52.505427 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.505397    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-data" (OuterVolumeSpecName: "istio-data") pod "48b7e9a7-5260-4dda-aba4-87908f97e48b" (UID: "48b7e9a7-5260-4dda-aba4-87908f97e48b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:22:52.506385 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.506362    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "48b7e9a7-5260-4dda-aba4-87908f97e48b" (UID: "48b7e9a7-5260-4dda-aba4-87908f97e48b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 22:22:52.506473 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.506411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-token" (OuterVolumeSpecName: "istio-token") pod "48b7e9a7-5260-4dda-aba4-87908f97e48b" (UID: "48b7e9a7-5260-4dda-aba4-87908f97e48b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:22:52.506928 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.506902    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "48b7e9a7-5260-4dda-aba4-87908f97e48b" (UID: "48b7e9a7-5260-4dda-aba4-87908f97e48b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:22:52.506993 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.506915    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-kube-api-access-g98qk" (OuterVolumeSpecName: "kube-api-access-g98qk") pod "48b7e9a7-5260-4dda-aba4-87908f97e48b" (UID: "48b7e9a7-5260-4dda-aba4-87908f97e48b"). InnerVolumeSpecName "kube-api-access-g98qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:22:52.605678 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.605643    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-podinfo\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:52.605678 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.605676    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b7e9a7-5260-4dda-aba4-87908f97e48b-istiod-ca-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:52.605922 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.605690    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-envoy\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:52.605922 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.605700    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g98qk\" (UniqueName: \"kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-kube-api-access-g98qk\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:52.605922 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.605711    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-workload-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:52.605922 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.605719    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-data\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:52.605922 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.605727    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b7e9a7-5260-4dda-aba4-87908f97e48b-istio-token\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:22:52.734048 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.734013    2579 generic.go:356] "Generic (PLEG): container finished" podID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerID="d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088" exitCode=0
Jul 23 22:22:52.734213 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.734063    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"
Jul 23 22:22:52.734213 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.734102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" event={"ID":"48b7e9a7-5260-4dda-aba4-87908f97e48b","Type":"ContainerDied","Data":"d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088"}
Jul 23 22:22:52.734213 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.734162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px" event={"ID":"48b7e9a7-5260-4dda-aba4-87908f97e48b","Type":"ContainerDied","Data":"f1891d3c5234acc206aaf074d88d8a3ccfdbdbded3ef558a6317f3d41977f946"}
Jul 23 22:22:52.734213 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.734184    2579 scope.go:117] "RemoveContainer" containerID="d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088"
Jul 23 22:22:52.742945 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.742928    2579 scope.go:117] "RemoveContainer" containerID="e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7"
Jul 23 22:22:52.750533 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.750515    2579 scope.go:117] "RemoveContainer" containerID="d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088"
Jul 23 22:22:52.750768 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:22:52.750746    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088\": container with ID starting with d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088 not found: ID does not exist" containerID="d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088"
Jul 23 22:22:52.750827 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.750778    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088"} err="failed to get container status \"d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088\": rpc error: code = NotFound desc = could not find container \"d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088\": container with ID starting with d56131eff63a3166a28d74224791b00ae41ef1dd80c6528088b497bf89348088 not found: ID does not exist"
Jul 23 22:22:52.750827 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.750796    2579 scope.go:117] "RemoveContainer" containerID="e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7"
Jul 23 22:22:52.751048 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:22:52.751033    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7\": container with ID starting with e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7 not found: ID does not exist" containerID="e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7"
Jul 23 22:22:52.751103 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.751051    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7"} err="failed to get container status \"e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7\": rpc error: code = NotFound desc = could not find container \"e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7\": container with ID starting with e7d5e440b1d5cfb86f8caeafb148ee016f919ce872d498a6b6638d57840927a7 not found: ID does not exist"
Jul 23 22:22:52.758628 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.758567    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"]
Jul 23 22:22:52.765237 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:52.765210    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1b6e6ae2-openshift-default-5f4b685d6c-gd8px"]
Jul 23 22:22:53.111566 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:53.111472    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" path="/var/lib/kubelet/pods/48b7e9a7-5260-4dda-aba4-87908f97e48b/volumes"
Jul 23 22:22:55.664641 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:55.664605    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:55.774756 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:55.774715    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:55.868375 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:55.868345    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:55.966739 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:55.966707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:22:56.398000 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:22:56.397913    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:13.453095 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:23:13.452910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:22.408791 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:23:22.408752    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:29.732182 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:23:29.732142    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:29.833643 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:23:29.833609    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:29.937042 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:23:29.937007    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:45.886887 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:23:45.886793    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:52.268347 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:23:52.268302    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:52.339224 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:23:52.339187    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:52.413634 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:23:52.413598    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:23:52.512986 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:23:52.512949    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:03.024639 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:03.024605    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:03.866434 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:03.866401    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:09.861006 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:09.860967    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:11.121470 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:11.121432    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:11.236052 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:11.236016    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:11.364153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:11.364117    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:11.485128 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:11.485097    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:12.335485 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:12.335448    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:12.454891 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:12.454810    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:12.611607 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:12.611520    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:14.539030 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:14.538982    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:21.009591 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:21.009552    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:21.105519 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:21.105483    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:21.735530 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:21.735494    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:28.941751 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:28.941696    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:29.010355 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.010321    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:29.200907 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.200839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:29.505617 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505544    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"]
Jul 23 22:24:29.505849 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505836    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4dca755-155f-426d-ad71-fc2e2eb867a8" containerName="revoke-keys"
Jul 23 22:24:29.505916 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505850    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4dca755-155f-426d-ad71-fc2e2eb867a8" containerName="revoke-keys"
Jul 23 22:24:29.505916 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505875    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerName="istio-proxy"
Jul 23 22:24:29.505916 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505880    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerName="istio-proxy"
Jul 23 22:24:29.505916 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505890    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerName="istio-proxy"
Jul 23 22:24:29.505916 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505897    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerName="istio-proxy"
Jul 23 22:24:29.505916 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505903    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f5726c-07f9-48ed-8ab6-632d52bba9e9" containerName="maas-api"
Jul 23 22:24:29.505916 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505908    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f5726c-07f9-48ed-8ab6-632d52bba9e9" containerName="maas-api"
Jul 23 22:24:29.506129 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505961    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f5726c-07f9-48ed-8ab6-632d52bba9e9" containerName="maas-api"
Jul 23 22:24:29.506129 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505969    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4dca755-155f-426d-ad71-fc2e2eb867a8" containerName="revoke-keys"
Jul 23 22:24:29.506129 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505975    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerName="istio-proxy"
Jul 23 22:24:29.506129 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.505982    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="48b7e9a7-5260-4dda-aba4-87908f97e48b" containerName="istio-proxy"
Jul 23 22:24:29.508890 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.508846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.511427 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.511390    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 22:24:29.511605 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.511587    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b9288e4f-openshift-default-dockercfg-xhxkb\""
Jul 23 22:24:29.533147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.533107    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"]
Jul 23 22:24:29.536184 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.536126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:29.562013 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.561978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-envoy\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.562197 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.562027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-podinfo\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.562197 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.562053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrc9\" (UniqueName: \"kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-kube-api-access-sxrc9\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.562197 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.562123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-token\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.562197 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.562181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-credential-socket\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.562338 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.562210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-certs\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.562338 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.562230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-socket\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.562338 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.562279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2314ac0a-fbd9-46ba-a972-405172f3884f-istiod-ca-cert\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.562338 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.562296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-data\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.636912 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.636880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:29.663209 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-podinfo\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.663393 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrc9\" (UniqueName: \"kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-kube-api-access-sxrc9\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.663393 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-token\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.663393 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-credential-socket\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.663393 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-certs\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.663393 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-socket\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.663657 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2314ac0a-fbd9-46ba-a972-405172f3884f-istiod-ca-cert\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.663657 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-data\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.663657 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-envoy\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.663819 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-credential-socket\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.663819 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-certs\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.663962 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-socket\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.664007 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.663980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-data\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.664183 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.664166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2314ac0a-fbd9-46ba-a972-405172f3884f-istiod-ca-cert\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.665825 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.665807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-podinfo\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.665944 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.665852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-envoy\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.671269 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.671244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-token\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.671394 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.671355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrc9\" (UniqueName: \"kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-kube-api-access-sxrc9\") pod \"e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") " pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.819710 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.819619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:29.961589 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.961556    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"]
Jul 23 22:24:29.964282 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:24:29.964244    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2314ac0a_fbd9_46ba_a972_405172f3884f.slice/crio-faf9edcf438d3d6a5802a36016655539f68e446574f878158a74b8ea0c929e09 WatchSource:0}: Error finding container faf9edcf438d3d6a5802a36016655539f68e446574f878158a74b8ea0c929e09: Status 404 returned error can't find the container with id faf9edcf438d3d6a5802a36016655539f68e446574f878158a74b8ea0c929e09
Jul 23 22:24:29.966284 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.966244    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:24:29.966384 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.966325    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:24:29.966384 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:29.966363    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:24:30.153797 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:30.153681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" event={"ID":"2314ac0a-fbd9-46ba-a972-405172f3884f","Type":"ContainerStarted","Data":"dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a"}
Jul 23 22:24:30.153797 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:30.153751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" event={"ID":"2314ac0a-fbd9-46ba-a972-405172f3884f","Type":"ContainerStarted","Data":"faf9edcf438d3d6a5802a36016655539f68e446574f878158a74b8ea0c929e09"}
Jul 23 22:24:30.181950 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:30.181894    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" podStartSLOduration=1.181847298 podStartE2EDuration="1.181847298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:24:30.181118952 +0000 UTC m=+1965.571511576" watchObservedRunningTime="2026-07-23 22:24:30.181847298 +0000 UTC m=+1965.572239915"
Jul 23 22:24:30.820568 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:30.820517    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:31.825135 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:31.825103    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:32.163568 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.163482    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:32.164682 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.164661    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:32.455161 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.455133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:32.554489 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.554454    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:32.694662 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.694628    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"]
Jul 23 22:24:32.698575 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.698555    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:32.701447 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.701426    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b9288e4f-serving-cert\""
Jul 23 22:24:32.710604 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.710582    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"]
Jul 23 22:24:32.732149 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.732108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:32.792490 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.792445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06bb427e-7439-4736-870d-e2605e64340f-openshift-service-ca\") pod \"maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:32.792683 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.792499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdwwg\" (UniqueName: \"kubernetes.io/projected/06bb427e-7439-4736-870d-e2605e64340f-kube-api-access-kdwwg\") pod \"maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:32.792683 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.792585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06bb427e-7439-4736-870d-e2605e64340f-maas-api-tls\") pod \"maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:32.893912 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.893827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06bb427e-7439-4736-870d-e2605e64340f-maas-api-tls\") pod \"maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:32.894400 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.893971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06bb427e-7439-4736-870d-e2605e64340f-openshift-service-ca\") pod \"maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:32.894400 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:24:32.893995    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-serving-cert: secret "maas-api-e2e-mt-b9288e4f-serving-cert" not found
Jul 23 22:24:32.894400 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.894009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdwwg\" (UniqueName: \"kubernetes.io/projected/06bb427e-7439-4736-870d-e2605e64340f-kube-api-access-kdwwg\") pod \"maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:32.894400 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:24:32.894083    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06bb427e-7439-4736-870d-e2605e64340f-maas-api-tls podName:06bb427e-7439-4736-870d-e2605e64340f nodeName:}" failed. No retries permitted until 2026-07-23 22:24:33.394064675 +0000 UTC m=+1968.784457271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06bb427e-7439-4736-870d-e2605e64340f-maas-api-tls") pod "maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q" (UID: "06bb427e-7439-4736-870d-e2605e64340f") : secret "maas-api-e2e-mt-b9288e4f-serving-cert" not found
Jul 23 22:24:32.894748 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.894730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06bb427e-7439-4736-870d-e2605e64340f-openshift-service-ca\") pod \"maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:32.905960 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:32.905931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdwwg\" (UniqueName: \"kubernetes.io/projected/06bb427e-7439-4736-870d-e2605e64340f-kube-api-access-kdwwg\") pod \"maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:33.206592 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:33.206558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:33.312883 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:33.312832    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:33.398795 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:33.398756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06bb427e-7439-4736-870d-e2605e64340f-maas-api-tls\") pod \"maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:33.401417 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:33.401386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06bb427e-7439-4736-870d-e2605e64340f-maas-api-tls\") pod \"maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:33.407369 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:33.407337    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:33.609813 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:33.609715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:33.749574 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:33.749549    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"]
Jul 23 22:24:33.751794 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:24:33.751768    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bb427e_7439_4736_870d_e2605e64340f.slice/crio-910ebd541f4df5169d288288f7ef886c9262d41b964db087a52954748ebe25c6 WatchSource:0}: Error finding container 910ebd541f4df5169d288288f7ef886c9262d41b964db087a52954748ebe25c6: Status 404 returned error can't find the container with id 910ebd541f4df5169d288288f7ef886c9262d41b964db087a52954748ebe25c6
Jul 23 22:24:34.173578 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:34.173533    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q" event={"ID":"06bb427e-7439-4736-870d-e2605e64340f","Type":"ContainerStarted","Data":"910ebd541f4df5169d288288f7ef886c9262d41b964db087a52954748ebe25c6"}
Jul 23 22:24:35.179096 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:35.179004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q" event={"ID":"06bb427e-7439-4736-870d-e2605e64340f","Type":"ContainerStarted","Data":"d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104"}
Jul 23 22:24:35.179498 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:35.179193    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:35.198490 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:35.198426    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q" podStartSLOduration=2.61836968 podStartE2EDuration="3.198405146s" totalImagesPullingTime="580.035466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:24:35.197401568 +0000 UTC m=+1970.587794177" watchObservedRunningTime="2026-07-23 22:24:35.198405146 +0000 UTC m=+1970.588797764"
Jul 23 22:24:36.972969 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:36.972909    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:37.662321 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:37.662284    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:37.748739 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:37.748705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:38.105944 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:38.105910    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:41.193823 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:41.193782    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:44.360128 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:44.360090    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:44.435395 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:44.435365    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:44.726906 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:44.726855    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"]
Jul 23 22:24:44.730121 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:44.730102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"
Jul 23 22:24:44.732303 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:44.732281    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lq4mm\""
Jul 23 22:24:44.738335 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:44.737370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"]
Jul 23 22:24:44.899679 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:44.899634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5335d1e3-090a-4b28-bd00-db030a5ea999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f\" (UID: \"5335d1e3-090a-4b28-bd00-db030a5ea999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"
Jul 23 22:24:45.000456 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:45.000381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5335d1e3-090a-4b28-bd00-db030a5ea999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f\" (UID: \"5335d1e3-090a-4b28-bd00-db030a5ea999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"
Jul 23 22:24:45.001024 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:45.001001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5335d1e3-090a-4b28-bd00-db030a5ea999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f\" (UID: \"5335d1e3-090a-4b28-bd00-db030a5ea999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"
Jul 23 22:24:45.045181 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:45.045145    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lq4mm\""
Jul 23 22:24:45.053107 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:45.053080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"
Jul 23 22:24:45.194740 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:45.194693    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"]
Jul 23 22:24:45.198715 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:24:45.198689    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5335d1e3_090a_4b28_bd00_db030a5ea999.slice/crio-eb9496d47be71e2f7c223256a232fd36794d147a987fd519c11a9125a4d44757 WatchSource:0}: Error finding container eb9496d47be71e2f7c223256a232fd36794d147a987fd519c11a9125a4d44757: Status 404 returned error can't find the container with id eb9496d47be71e2f7c223256a232fd36794d147a987fd519c11a9125a4d44757
Jul 23 22:24:45.230401 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:45.230369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f" event={"ID":"5335d1e3-090a-4b28-bd00-db030a5ea999","Type":"ContainerStarted","Data":"eb9496d47be71e2f7c223256a232fd36794d147a987fd519c11a9125a4d44757"}
Jul 23 22:24:46.236557 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:46.236519    2579 generic.go:356] "Generic (PLEG): container finished" podID="5335d1e3-090a-4b28-bd00-db030a5ea999" containerID="d494ba190007af8ec9739cbfc2a46a42791bedd63befef0e66dd5e79ee58e405" exitCode=0
Jul 23 22:24:46.237057 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:46.236604    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f" event={"ID":"5335d1e3-090a-4b28-bd00-db030a5ea999","Type":"ContainerDied","Data":"d494ba190007af8ec9739cbfc2a46a42791bedd63befef0e66dd5e79ee58e405"}
Jul 23 22:24:46.238082 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:46.238060    2579 generic.go:356] "Generic (PLEG): container finished" podID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerID="dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a" exitCode=0
Jul 23 22:24:46.238203 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:46.238136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" event={"ID":"2314ac0a-fbd9-46ba-a972-405172f3884f","Type":"ContainerDied","Data":"dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a"}
Jul 23 22:24:46.238502 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:46.238488    2579 scope.go:117] "RemoveContainer" containerID="dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a"
Jul 23 22:24:46.239450 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:46.239407    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:24:46.239531 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:46.239485    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:24:46.239531 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:46.239515    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:24:47.164336 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:47.164304    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:47.245108 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:47.245064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" event={"ID":"2314ac0a-fbd9-46ba-a972-405172f3884f","Type":"ContainerStarted","Data":"2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041"}
Jul 23 22:24:47.422220 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:47.422151    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"
Jul 23 22:24:47.521457 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:47.521427    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5335d1e3-090a-4b28-bd00-db030a5ea999-maas-api-service-ca\") pod \"5335d1e3-090a-4b28-bd00-db030a5ea999\" (UID: \"5335d1e3-090a-4b28-bd00-db030a5ea999\") "
Jul 23 22:24:47.521829 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:47.521802    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5335d1e3-090a-4b28-bd00-db030a5ea999-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5335d1e3-090a-4b28-bd00-db030a5ea999" (UID: "5335d1e3-090a-4b28-bd00-db030a5ea999"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:24:47.622929 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:47.622886    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5335d1e3-090a-4b28-bd00-db030a5ea999-maas-api-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:24:47.820162 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:47.820119    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:47.821460 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:47.821429    2579 patch_prober.go:28] interesting pod/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 23 22:24:47.821596 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:47.821489    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 23 22:24:48.251086 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:48.251051    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"
Jul 23 22:24:48.251513 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:48.251086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f" event={"ID":"5335d1e3-090a-4b28-bd00-db030a5ea999","Type":"ContainerDied","Data":"eb9496d47be71e2f7c223256a232fd36794d147a987fd519c11a9125a4d44757"}
Jul 23 22:24:48.251513 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:48.251146    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb9496d47be71e2f7c223256a232fd36794d147a987fd519c11a9125a4d44757"
Jul 23 22:24:48.820726 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:48.820689    2579 patch_prober.go:28] interesting pod/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 23 22:24:48.820944 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:48.820744    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 23 22:24:49.820254 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:49.820219    2579 patch_prober.go:28] interesting pod/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 23 22:24:49.820678 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:49.820292    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 23 22:24:50.820844 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:50.820806    2579 patch_prober.go:28] interesting pod/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Jul 23 22:24:50.821380 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:50.820894    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Jul 23 22:24:51.281409 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:51.281296    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:51.354511 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:51.354470    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:51.825306 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:51.825275    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:51.825739 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:51.825616    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:51.826450 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:51.826431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:24:55.189406 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.189365    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"]
Jul 23 22:24:55.189899 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.189582    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q" podUID="06bb427e-7439-4736-870d-e2605e64340f" containerName="maas-api" containerID="cri-o://d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104" gracePeriod=30
Jul 23 22:24:55.219445 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.219389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:55.438682 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.438657    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:55.593514 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.593477    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06bb427e-7439-4736-870d-e2605e64340f-openshift-service-ca\") pod \"06bb427e-7439-4736-870d-e2605e64340f\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") "
Jul 23 22:24:55.593514 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.593527    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06bb427e-7439-4736-870d-e2605e64340f-maas-api-tls\") pod \"06bb427e-7439-4736-870d-e2605e64340f\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") "
Jul 23 22:24:55.593796 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.593583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdwwg\" (UniqueName: \"kubernetes.io/projected/06bb427e-7439-4736-870d-e2605e64340f-kube-api-access-kdwwg\") pod \"06bb427e-7439-4736-870d-e2605e64340f\" (UID: \"06bb427e-7439-4736-870d-e2605e64340f\") "
Jul 23 22:24:55.593911 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.593884    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bb427e-7439-4736-870d-e2605e64340f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06bb427e-7439-4736-870d-e2605e64340f" (UID: "06bb427e-7439-4736-870d-e2605e64340f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:24:55.595758 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.595730    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bb427e-7439-4736-870d-e2605e64340f-kube-api-access-kdwwg" (OuterVolumeSpecName: "kube-api-access-kdwwg") pod "06bb427e-7439-4736-870d-e2605e64340f" (UID: "06bb427e-7439-4736-870d-e2605e64340f"). InnerVolumeSpecName "kube-api-access-kdwwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:24:55.595884 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.595801    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06bb427e-7439-4736-870d-e2605e64340f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06bb427e-7439-4736-870d-e2605e64340f" (UID: "06bb427e-7439-4736-870d-e2605e64340f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:24:55.695015 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.694978    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdwwg\" (UniqueName: \"kubernetes.io/projected/06bb427e-7439-4736-870d-e2605e64340f-kube-api-access-kdwwg\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:24:55.695015 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.695012    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06bb427e-7439-4736-870d-e2605e64340f-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:24:55.695219 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:55.695026    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06bb427e-7439-4736-870d-e2605e64340f-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:24:56.291750 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:56.291715    2579 generic.go:356] "Generic (PLEG): container finished" podID="06bb427e-7439-4736-870d-e2605e64340f" containerID="d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104" exitCode=0
Jul 23 22:24:56.292219 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:56.291758    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"
Jul 23 22:24:56.292219 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:56.291796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q" event={"ID":"06bb427e-7439-4736-870d-e2605e64340f","Type":"ContainerDied","Data":"d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104"}
Jul 23 22:24:56.292219 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:56.291851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q" event={"ID":"06bb427e-7439-4736-870d-e2605e64340f","Type":"ContainerDied","Data":"910ebd541f4df5169d288288f7ef886c9262d41b964db087a52954748ebe25c6"}
Jul 23 22:24:56.292219 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:56.291895    2579 scope.go:117] "RemoveContainer" containerID="d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104"
Jul 23 22:24:56.300916 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:56.300897    2579 scope.go:117] "RemoveContainer" containerID="d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104"
Jul 23 22:24:56.301223 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:24:56.301201    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104\": container with ID starting with d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104 not found: ID does not exist" containerID="d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104"
Jul 23 22:24:56.301304 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:56.301234    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104"} err="failed to get container status \"d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104\": rpc error: code = NotFound desc = could not find container \"d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104\": container with ID starting with d5007756f35a3dcf71b7865f43b84bb1fa6e68799201694db8fda9fe3d536104 not found: ID does not exist"
Jul 23 22:24:56.312944 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:56.312914    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"]
Jul 23 22:24:56.317288 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:56.317261    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b9288e4f-6d84ccd7fc-cvl7q"]
Jul 23 22:24:57.111041 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:57.111005    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06bb427e-7439-4736-870d-e2605e64340f" path="/var/lib/kubelet/pods/06bb427e-7439-4736-870d-e2605e64340f/volumes"
Jul 23 22:24:59.809264 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:59.809230    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"]
Jul 23 22:24:59.809746 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:59.809703    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:24:59.812246 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:24:59.812222    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b9288e4f-txk4f"]
Jul 23 22:25:00.201733 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:00.201705    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:01.111177 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:01.111139    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5335d1e3-090a-4b28-bd00-db030a5ea999" path="/var/lib/kubelet/pods/5335d1e3-090a-4b28-bd00-db030a5ea999/volumes"
Jul 23 22:25:06.601400 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:06.601360    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:06.679087 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:06.679053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:06.895242 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:06.895132    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"]
Jul 23 22:25:06.895417 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:06.895347    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerName="istio-proxy" containerID="cri-o://2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041" gracePeriod=30
Jul 23 22:25:06.915339 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:06.915307    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:07.161969 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:07.161886    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:07.301532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:07.301495    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:07.467215 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:07.467184    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:07.613423 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:07.613391    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:12.140755 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.140727    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:25:12.233249 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233210    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-envoy\") pod \"2314ac0a-fbd9-46ba-a972-405172f3884f\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") "
Jul 23 22:25:12.233459 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233283    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxrc9\" (UniqueName: \"kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-kube-api-access-sxrc9\") pod \"2314ac0a-fbd9-46ba-a972-405172f3884f\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") "
Jul 23 22:25:12.233459 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233316    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-data\") pod \"2314ac0a-fbd9-46ba-a972-405172f3884f\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") "
Jul 23 22:25:12.233459 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233348    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-socket\") pod \"2314ac0a-fbd9-46ba-a972-405172f3884f\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") "
Jul 23 22:25:12.233459 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233374    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-token\") pod \"2314ac0a-fbd9-46ba-a972-405172f3884f\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") "
Jul 23 22:25:12.233459 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-certs\") pod \"2314ac0a-fbd9-46ba-a972-405172f3884f\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") "
Jul 23 22:25:12.233459 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233440    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-credential-socket\") pod \"2314ac0a-fbd9-46ba-a972-405172f3884f\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") "
Jul 23 22:25:12.233760 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233470    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2314ac0a-fbd9-46ba-a972-405172f3884f-istiod-ca-cert\") pod \"2314ac0a-fbd9-46ba-a972-405172f3884f\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") "
Jul 23 22:25:12.233760 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233519    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-podinfo\") pod \"2314ac0a-fbd9-46ba-a972-405172f3884f\" (UID: \"2314ac0a-fbd9-46ba-a972-405172f3884f\") "
Jul 23 22:25:12.233760 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233674    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2314ac0a-fbd9-46ba-a972-405172f3884f" (UID: "2314ac0a-fbd9-46ba-a972-405172f3884f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:25:12.233972 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233822    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:25:12.233972 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.233940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2314ac0a-fbd9-46ba-a972-405172f3884f" (UID: "2314ac0a-fbd9-46ba-a972-405172f3884f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:25:12.234094 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.234036    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2314ac0a-fbd9-46ba-a972-405172f3884f" (UID: "2314ac0a-fbd9-46ba-a972-405172f3884f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:25:12.234153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.234112    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2314ac0a-fbd9-46ba-a972-405172f3884f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2314ac0a-fbd9-46ba-a972-405172f3884f" (UID: "2314ac0a-fbd9-46ba-a972-405172f3884f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:25:12.234459 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.234406    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-data" (OuterVolumeSpecName: "istio-data") pod "2314ac0a-fbd9-46ba-a972-405172f3884f" (UID: "2314ac0a-fbd9-46ba-a972-405172f3884f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:25:12.235658 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.235629    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-kube-api-access-sxrc9" (OuterVolumeSpecName: "kube-api-access-sxrc9") pod "2314ac0a-fbd9-46ba-a972-405172f3884f" (UID: "2314ac0a-fbd9-46ba-a972-405172f3884f"). InnerVolumeSpecName "kube-api-access-sxrc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:25:12.235802 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.235781    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2314ac0a-fbd9-46ba-a972-405172f3884f" (UID: "2314ac0a-fbd9-46ba-a972-405172f3884f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 22:25:12.236130 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.236099    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2314ac0a-fbd9-46ba-a972-405172f3884f" (UID: "2314ac0a-fbd9-46ba-a972-405172f3884f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:25:12.236130 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.236102    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-token" (OuterVolumeSpecName: "istio-token") pod "2314ac0a-fbd9-46ba-a972-405172f3884f" (UID: "2314ac0a-fbd9-46ba-a972-405172f3884f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:25:12.334579 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.334537    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxrc9\" (UniqueName: \"kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-kube-api-access-sxrc9\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:25:12.334579 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.334573    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-data\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:25:12.334579 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.334582    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-token\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:25:12.334579 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.334590    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-workload-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:25:12.334851 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.334598    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-credential-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:25:12.334851 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.334606    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2314ac0a-fbd9-46ba-a972-405172f3884f-istiod-ca-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:25:12.334851 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.334613    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-podinfo\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:25:12.334851 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.334620    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2314ac0a-fbd9-46ba-a972-405172f3884f-istio-envoy\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:25:12.366481 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.366440    2579 generic.go:356] "Generic (PLEG): container finished" podID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerID="2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041" exitCode=0
Jul 23 22:25:12.366664 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.366501    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"
Jul 23 22:25:12.366664 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.366524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" event={"ID":"2314ac0a-fbd9-46ba-a972-405172f3884f","Type":"ContainerDied","Data":"2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041"}
Jul 23 22:25:12.366664 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.366610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp" event={"ID":"2314ac0a-fbd9-46ba-a972-405172f3884f","Type":"ContainerDied","Data":"faf9edcf438d3d6a5802a36016655539f68e446574f878158a74b8ea0c929e09"}
Jul 23 22:25:12.366664 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.366638    2579 scope.go:117] "RemoveContainer" containerID="2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041"
Jul 23 22:25:12.375523 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.375506    2579 scope.go:117] "RemoveContainer" containerID="dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a"
Jul 23 22:25:12.383280 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.383263    2579 scope.go:117] "RemoveContainer" containerID="2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041"
Jul 23 22:25:12.383549 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:25:12.383529    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041\": container with ID starting with 2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041 not found: ID does not exist" containerID="2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041"
Jul 23 22:25:12.383588 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.383559    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041"} err="failed to get container status \"2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041\": rpc error: code = NotFound desc = could not find container \"2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041\": container with ID starting with 2474393b1bcdcc042bb8152bc70185a06106e9b6cfa14fa6e4e170f147756041 not found: ID does not exist"
Jul 23 22:25:12.383588 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.383579    2579 scope.go:117] "RemoveContainer" containerID="dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a"
Jul 23 22:25:12.383806 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:25:12.383790    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a\": container with ID starting with dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a not found: ID does not exist" containerID="dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a"
Jul 23 22:25:12.383843 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.383810    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a"} err="failed to get container status \"dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a\": rpc error: code = NotFound desc = could not find container \"dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a\": container with ID starting with dc36ce17edd7357be0709b851762ecd5a8ce08b6c7adb1bb9464149c509f614a not found: ID does not exist"
Jul 23 22:25:12.389037 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.389012    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"]
Jul 23 22:25:12.395282 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:12.395253    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b9288e4f-openshift-default-96f8d97f5-nsvwp"]
Jul 23 22:25:13.111618 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:13.111585    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" path="/var/lib/kubelet/pods/2314ac0a-fbd9-46ba-a972-405172f3884f/volumes"
Jul 23 22:25:13.582724 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:13.582681    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:13.692064 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:13.692026    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:13.791260 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:13.791227    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:13.950887 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:13.950837    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:20.101536 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:20.101496    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:20.202272 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:20.202231    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:27.056004 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.055970    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:27.358603 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358521    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"]
Jul 23 22:25:27.358834 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358820    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5335d1e3-090a-4b28-bd00-db030a5ea999" containerName="revoke-keys"
Jul 23 22:25:27.358921 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358836    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5335d1e3-090a-4b28-bd00-db030a5ea999" containerName="revoke-keys"
Jul 23 22:25:27.358921 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358846    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerName="istio-proxy"
Jul 23 22:25:27.358921 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358852    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerName="istio-proxy"
Jul 23 22:25:27.358921 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358876    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerName="istio-proxy"
Jul 23 22:25:27.358921 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358883    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerName="istio-proxy"
Jul 23 22:25:27.358921 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358898    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06bb427e-7439-4736-870d-e2605e64340f" containerName="maas-api"
Jul 23 22:25:27.358921 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358903    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="06bb427e-7439-4736-870d-e2605e64340f" containerName="maas-api"
Jul 23 22:25:27.359142 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358955    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerName="istio-proxy"
Jul 23 22:25:27.359142 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358962    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2314ac0a-fbd9-46ba-a972-405172f3884f" containerName="istio-proxy"
Jul 23 22:25:27.359142 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358968    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="06bb427e-7439-4736-870d-e2605e64340f" containerName="maas-api"
Jul 23 22:25:27.359142 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.358973    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5335d1e3-090a-4b28-bd00-db030a5ea999" containerName="revoke-keys"
Jul 23 22:25:27.361075 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.361054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.363362 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.363342    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-82711a26-openshift-default-dockercfg-l4fkd\""
Jul 23 22:25:27.363605 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.363585    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 22:25:27.377322 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.377292    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"]
Jul 23 22:25:27.389327 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.389300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:27.463543 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.463499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmvs7\" (UniqueName: \"kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-kube-api-access-xmvs7\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.463720 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.463556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2dbe01d-81fb-4f03-be67-3c00938fd728-istiod-ca-cert\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.463720 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.463635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-podinfo\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.463720 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.463663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-token\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.463720 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.463688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-certs\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.463896 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.463732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-envoy\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.463896 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.463762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-data\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.463896 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.463819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-socket\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.463896 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.463848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-credential-socket\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.564587 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.564548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-token\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.564758 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.564593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-certs\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.564758 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.564635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-envoy\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.564758 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.564656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-data\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.564758 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.564711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-socket\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.564758 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.564747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-credential-socket\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.565078 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.564782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmvs7\" (UniqueName: \"kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-kube-api-access-xmvs7\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.565078 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.564822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2dbe01d-81fb-4f03-be67-3c00938fd728-istiod-ca-cert\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.565078 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.564919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-podinfo\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.565078 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.565044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-certs\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.565290 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.565077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-socket\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.565290 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.565148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-data\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.565290 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.565227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-credential-socket\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.565666 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.565647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2dbe01d-81fb-4f03-be67-3c00938fd728-istiod-ca-cert\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.567142 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.567115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-envoy\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.567362 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.567344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-podinfo\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.573589 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.573562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-token\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.573713 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.573630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmvs7\" (UniqueName: \"kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-kube-api-access-xmvs7\") pod \"e2e-mt-82711a26-openshift-default-79f75d87bc-cq698\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") " pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.671381 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.671295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:27.777347 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.777289    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:27.811220 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.811191    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"]
Jul 23 22:25:27.813320 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:25:27.813294    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2dbe01d_81fb_4f03_be67_3c00938fd728.slice/crio-2901928f7cee0665ef2f0056ad2d6e6724cf0c4488e39f75be0b3e5862148e27 WatchSource:0}: Error finding container 2901928f7cee0665ef2f0056ad2d6e6724cf0c4488e39f75be0b3e5862148e27: Status 404 returned error can't find the container with id 2901928f7cee0665ef2f0056ad2d6e6724cf0c4488e39f75be0b3e5862148e27
Jul 23 22:25:27.815438 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.815406    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:25:27.815525 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.815497    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:25:27.815563 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:27.815527    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:25:28.446553 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:28.446516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" event={"ID":"e2dbe01d-81fb-4f03-be67-3c00938fd728","Type":"ContainerStarted","Data":"7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b"}
Jul 23 22:25:28.446997 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:28.446574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" event={"ID":"e2dbe01d-81fb-4f03-be67-3c00938fd728","Type":"ContainerStarted","Data":"2901928f7cee0665ef2f0056ad2d6e6724cf0c4488e39f75be0b3e5862148e27"}
Jul 23 22:25:28.466292 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:28.466227    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" podStartSLOduration=1.4662049750000001 podStartE2EDuration="1.466204975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:25:28.465253714 +0000 UTC m=+2023.855646330" watchObservedRunningTime="2026-07-23 22:25:28.466204975 +0000 UTC m=+2023.856597576"
Jul 23 22:25:28.672297 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:28.672250    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:28.673844 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:28.673817    2579 patch_prober.go:28] interesting pod/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 23 22:25:28.673916 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:28.673885    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 23 22:25:29.676101 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:29.676067    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:30.448708 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.448671    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:30.455347 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.455324    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:30.456375 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.456358    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:30.544000 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.543946    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:30.692555 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.692517    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"]
Jul 23 22:25:30.695277 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.695259    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:30.697583 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.697560    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-82711a26-serving-cert\""
Jul 23 22:25:30.706827 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.706801    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"]
Jul 23 22:25:30.728303 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.728260    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:30.793580 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.793540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d1607a7-4ae5-484e-993f-73f3b2b978e9-maas-api-tls\") pod \"maas-api-e2e-mt-82711a26-77889cb55f-nqs9j\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:30.793763 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.793594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1607a7-4ae5-484e-993f-73f3b2b978e9-openshift-service-ca\") pod \"maas-api-e2e-mt-82711a26-77889cb55f-nqs9j\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:30.793763 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.793657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cknwh\" (UniqueName: \"kubernetes.io/projected/4d1607a7-4ae5-484e-993f-73f3b2b978e9-kube-api-access-cknwh\") pod \"maas-api-e2e-mt-82711a26-77889cb55f-nqs9j\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:30.827062 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.827027    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:30.894369 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.894323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cknwh\" (UniqueName: \"kubernetes.io/projected/4d1607a7-4ae5-484e-993f-73f3b2b978e9-kube-api-access-cknwh\") pod \"maas-api-e2e-mt-82711a26-77889cb55f-nqs9j\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:30.894545 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.894408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d1607a7-4ae5-484e-993f-73f3b2b978e9-maas-api-tls\") pod \"maas-api-e2e-mt-82711a26-77889cb55f-nqs9j\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:30.894545 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.894444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1607a7-4ae5-484e-993f-73f3b2b978e9-openshift-service-ca\") pod \"maas-api-e2e-mt-82711a26-77889cb55f-nqs9j\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:30.895136 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.895116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1607a7-4ae5-484e-993f-73f3b2b978e9-openshift-service-ca\") pod \"maas-api-e2e-mt-82711a26-77889cb55f-nqs9j\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:30.897107 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.897086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d1607a7-4ae5-484e-993f-73f3b2b978e9-maas-api-tls\") pod \"maas-api-e2e-mt-82711a26-77889cb55f-nqs9j\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:30.902688 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:30.902647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cknwh\" (UniqueName: \"kubernetes.io/projected/4d1607a7-4ae5-484e-993f-73f3b2b978e9-kube-api-access-cknwh\") pod \"maas-api-e2e-mt-82711a26-77889cb55f-nqs9j\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:31.006583 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:31.006491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:31.148005 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:31.147971    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"]
Jul 23 22:25:31.150559 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:25:31.150533    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1607a7_4ae5_484e_993f_73f3b2b978e9.slice/crio-da8a604d3137601ca8012e7e88851561549cdb7ac59a90969174fa4e02602538 WatchSource:0}: Error finding container da8a604d3137601ca8012e7e88851561549cdb7ac59a90969174fa4e02602538: Status 404 returned error can't find the container with id da8a604d3137601ca8012e7e88851561549cdb7ac59a90969174fa4e02602538
Jul 23 22:25:31.231646 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:31.231612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:31.377137 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:31.377053    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:31.460672 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:31.460635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j" event={"ID":"4d1607a7-4ae5-484e-993f-73f3b2b978e9","Type":"ContainerStarted","Data":"da8a604d3137601ca8012e7e88851561549cdb7ac59a90969174fa4e02602538"}
Jul 23 22:25:32.466421 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:32.466383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j" event={"ID":"4d1607a7-4ae5-484e-993f-73f3b2b978e9","Type":"ContainerStarted","Data":"6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b"}
Jul 23 22:25:32.466938 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:32.466542    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:32.483081 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:32.483005    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j" podStartSLOduration=1.853036079 podStartE2EDuration="2.482984792s" totalImagesPullingTime="629.948713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:25:32.482349076 +0000 UTC m=+2027.872741692" watchObservedRunningTime="2026-07-23 22:25:32.482984792 +0000 UTC m=+2027.873377424"
Jul 23 22:25:34.966107 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:34.966068    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:38.476837 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:38.476809    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:25:45.783257 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:45.783143    2579 patch_prober.go:28] interesting pod/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:57370->10.134.0.44:15021: read: connection reset by peer" start-of-body=
Jul 23 22:25:45.783257 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:45.783227    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:57370->10.134.0.44:15021: read: connection reset by peer"
Jul 23 22:25:46.534406 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:46.534372    2579 generic.go:356] "Generic (PLEG): container finished" podID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerID="7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b" exitCode=0
Jul 23 22:25:46.534597 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:46.534435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" event={"ID":"e2dbe01d-81fb-4f03-be67-3c00938fd728","Type":"ContainerDied","Data":"7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b"}
Jul 23 22:25:46.534846 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:46.534830    2579 scope.go:117] "RemoveContainer" containerID="7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b"
Jul 23 22:25:46.535812 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:46.535783    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:25:46.535885 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:46.535856    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:25:46.535939 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:46.535910    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:25:47.540569 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:47.540535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" event={"ID":"e2dbe01d-81fb-4f03-be67-3c00938fd728","Type":"ContainerStarted","Data":"e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75"}
Jul 23 22:25:47.672245 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:47.672206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:47.672658 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:47.672575    2579 patch_prober.go:28] interesting pod/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 23 22:25:47.672852 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:47.672830    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 23 22:25:48.672583 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:48.672537    2579 patch_prober.go:28] interesting pod/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 23 22:25:48.673235 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:48.673194    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 23 22:25:49.687199 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:49.687147    2579 patch_prober.go:28] interesting pod/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 23 22:25:49.687712 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:49.687224    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 23 22:25:50.672734 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:50.672679    2579 patch_prober.go:28] interesting pod/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 23 22:25:50.672993 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:50.672760    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 23 22:25:51.675088 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.675057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:51.675494 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.675426    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:51.676080 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.676055    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:25:51.721032 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.721001    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"]
Jul 23 22:25:51.724260 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.724235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.726510 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.726488    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-82711a26\"/\"default-dockercfg-rznk6\""
Jul 23 22:25:51.726510 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.726505    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-82711a26\"/\"kube-root-ca.crt\""
Jul 23 22:25:51.727031 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.727011    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-82711a26\"/\"openshift-service-ca.crt\""
Jul 23 22:25:51.727101 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.727033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-82711a26\"/\"e2e-lifecycle-model-82711a26-kserve-self-signed-certs\""
Jul 23 22:25:51.739021 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.738991    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"]
Jul 23 22:25:51.750925 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.750855    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:25:51.808885 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.808814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-home\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.808885 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.808887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-dshm\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.809152 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.808912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tls-certs\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.809152 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.809002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mtj\" (UniqueName: \"kubernetes.io/projected/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-kube-api-access-z6mtj\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.809152 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.809044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tmp-dir\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.809152 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.809111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-model-cache\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.909793 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.909743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-home\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.910022 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.909807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-dshm\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.910022 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.909930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tls-certs\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.910022 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.909990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mtj\" (UniqueName: \"kubernetes.io/projected/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-kube-api-access-z6mtj\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.910022 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.910015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tmp-dir\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.910247 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.910056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-model-cache\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.910320 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.910302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-home\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.910379 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.910351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-model-cache\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.910491 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.910466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tmp-dir\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.912335 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.912317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-dshm\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.912464 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.912448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tls-certs\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:51.918005 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:51.917984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mtj\" (UniqueName: \"kubernetes.io/projected/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-kube-api-access-z6mtj\") pod \"e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") " pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:52.033998 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:52.033957    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:25:52.176124 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:52.176088    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"]
Jul 23 22:25:52.177817 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:25:52.177789    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1048f8a_d3c8_446e_98e7_b1dffa54eba4.slice/crio-8564cdaf28d823298a3a26abf64664ee09e1755320e488d728cf69be6ad5e11d WatchSource:0}: Error finding container 8564cdaf28d823298a3a26abf64664ee09e1755320e488d728cf69be6ad5e11d: Status 404 returned error can't find the container with id 8564cdaf28d823298a3a26abf64664ee09e1755320e488d728cf69be6ad5e11d
Jul 23 22:25:52.563220 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:52.563182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h" event={"ID":"a1048f8a-d3c8-446e-98e7-b1dffa54eba4","Type":"ContainerStarted","Data":"e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e"}
Jul 23 22:25:52.563220 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:52.563228    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h" event={"ID":"a1048f8a-d3c8-446e-98e7-b1dffa54eba4","Type":"ContainerStarted","Data":"8564cdaf28d823298a3a26abf64664ee09e1755320e488d728cf69be6ad5e11d"}
Jul 23 22:25:52.579791 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:25:52.579728    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h" podStartSLOduration=1.579711632 podStartE2EDuration="1.579711632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:25:52.578257139 +0000 UTC m=+2047.968649757" watchObservedRunningTime="2026-07-23 22:25:52.579711632 +0000 UTC m=+2047.970104276"
Jul 23 22:26:02.034111 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:02.034071    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:26:02.034564 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:02.034386    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:26:02.047097 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:02.047067    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:26:02.619258 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:02.619223    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:26:18.740287 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:18.740199    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"]
Jul 23 22:26:18.743276 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:18.743256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"
Jul 23 22:26:18.745643 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:18.745606    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lq4mm\""
Jul 23 22:26:18.751418 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:18.751390    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"]
Jul 23 22:26:18.854594 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:18.854557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122be7a3-2a68-4ac5-9af6-360894a68a55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-82711a26-gfntk\" (UID: \"122be7a3-2a68-4ac5-9af6-360894a68a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"
Jul 23 22:26:18.955852 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:18.955809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122be7a3-2a68-4ac5-9af6-360894a68a55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-82711a26-gfntk\" (UID: \"122be7a3-2a68-4ac5-9af6-360894a68a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"
Jul 23 22:26:18.956480 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:18.956457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122be7a3-2a68-4ac5-9af6-360894a68a55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-82711a26-gfntk\" (UID: \"122be7a3-2a68-4ac5-9af6-360894a68a55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"
Jul 23 22:26:19.055024 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:19.054936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"
Jul 23 22:26:19.196026 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:19.195892    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"]
Jul 23 22:26:19.199854 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:26:19.199818    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122be7a3_2a68_4ac5_9af6_360894a68a55.slice/crio-2d68832c9031ae91a71c2ff939a532e10d0b9e3079e86e17b03d8d35fb763f56 WatchSource:0}: Error finding container 2d68832c9031ae91a71c2ff939a532e10d0b9e3079e86e17b03d8d35fb763f56: Status 404 returned error can't find the container with id 2d68832c9031ae91a71c2ff939a532e10d0b9e3079e86e17b03d8d35fb763f56
Jul 23 22:26:19.683534 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:19.683444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk" event={"ID":"122be7a3-2a68-4ac5-9af6-360894a68a55","Type":"ContainerStarted","Data":"ee5d61cebb7944897971fd5db9d7a20936c5f18491a8a08262a54e504df7dc7f"}
Jul 23 22:26:19.683534 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:19.683507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk" event={"ID":"122be7a3-2a68-4ac5-9af6-360894a68a55","Type":"ContainerStarted","Data":"2d68832c9031ae91a71c2ff939a532e10d0b9e3079e86e17b03d8d35fb763f56"}
Jul 23 22:26:19.701149 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:19.701098    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk" podStartSLOduration=1.701078811 podStartE2EDuration="1.701078811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:26:19.699163086 +0000 UTC m=+2075.089555695" watchObservedRunningTime="2026-07-23 22:26:19.701078811 +0000 UTC m=+2075.091471465"
Jul 23 22:26:20.689962 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:20.689926    2579 generic.go:356] "Generic (PLEG): container finished" podID="122be7a3-2a68-4ac5-9af6-360894a68a55" containerID="ee5d61cebb7944897971fd5db9d7a20936c5f18491a8a08262a54e504df7dc7f" exitCode=0
Jul 23 22:26:20.690346 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:20.690014    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk" event={"ID":"122be7a3-2a68-4ac5-9af6-360894a68a55","Type":"ContainerDied","Data":"ee5d61cebb7944897971fd5db9d7a20936c5f18491a8a08262a54e504df7dc7f"}
Jul 23 22:26:21.831312 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:21.831282    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"
Jul 23 22:26:21.979941 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:21.979910    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122be7a3-2a68-4ac5-9af6-360894a68a55-maas-api-service-ca\") pod \"122be7a3-2a68-4ac5-9af6-360894a68a55\" (UID: \"122be7a3-2a68-4ac5-9af6-360894a68a55\") "
Jul 23 22:26:21.980313 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:21.980288    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122be7a3-2a68-4ac5-9af6-360894a68a55-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "122be7a3-2a68-4ac5-9af6-360894a68a55" (UID: "122be7a3-2a68-4ac5-9af6-360894a68a55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:26:22.081201 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:22.081163    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/122be7a3-2a68-4ac5-9af6-360894a68a55-maas-api-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:26:22.707791 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:22.707756    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"
Jul 23 22:26:22.708018 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:22.707748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk" event={"ID":"122be7a3-2a68-4ac5-9af6-360894a68a55","Type":"ContainerDied","Data":"2d68832c9031ae91a71c2ff939a532e10d0b9e3079e86e17b03d8d35fb763f56"}
Jul 23 22:26:22.708018 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:22.707932    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d68832c9031ae91a71c2ff939a532e10d0b9e3079e86e17b03d8d35fb763f56"
Jul 23 22:26:34.236250 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.236216    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"]
Jul 23 22:26:34.236752 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.236431    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j" podUID="4d1607a7-4ae5-484e-993f-73f3b2b978e9" containerName="maas-api" containerID="cri-o://6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b" gracePeriod=30
Jul 23 22:26:34.270111 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.270075    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:26:34.484406 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.484380    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:26:34.591501 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.591406    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1607a7-4ae5-484e-993f-73f3b2b978e9-openshift-service-ca\") pod \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") "
Jul 23 22:26:34.591501 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.591472    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d1607a7-4ae5-484e-993f-73f3b2b978e9-maas-api-tls\") pod \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") "
Jul 23 22:26:34.591738 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.591604    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cknwh\" (UniqueName: \"kubernetes.io/projected/4d1607a7-4ae5-484e-993f-73f3b2b978e9-kube-api-access-cknwh\") pod \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\" (UID: \"4d1607a7-4ae5-484e-993f-73f3b2b978e9\") "
Jul 23 22:26:34.591892 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.591840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1607a7-4ae5-484e-993f-73f3b2b978e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d1607a7-4ae5-484e-993f-73f3b2b978e9" (UID: "4d1607a7-4ae5-484e-993f-73f3b2b978e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:26:34.593791 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.593770    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d1607a7-4ae5-484e-993f-73f3b2b978e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d1607a7-4ae5-484e-993f-73f3b2b978e9" (UID: "4d1607a7-4ae5-484e-993f-73f3b2b978e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:26:34.593891 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.593794    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1607a7-4ae5-484e-993f-73f3b2b978e9-kube-api-access-cknwh" (OuterVolumeSpecName: "kube-api-access-cknwh") pod "4d1607a7-4ae5-484e-993f-73f3b2b978e9" (UID: "4d1607a7-4ae5-484e-993f-73f3b2b978e9"). InnerVolumeSpecName "kube-api-access-cknwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:26:34.693251 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.693210    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cknwh\" (UniqueName: \"kubernetes.io/projected/4d1607a7-4ae5-484e-993f-73f3b2b978e9-kube-api-access-cknwh\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:26:34.693251 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.693247    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1607a7-4ae5-484e-993f-73f3b2b978e9-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:26:34.693251 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.693261    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d1607a7-4ae5-484e-993f-73f3b2b978e9-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:26:34.763531 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.763498    2579 generic.go:356] "Generic (PLEG): container finished" podID="4d1607a7-4ae5-484e-993f-73f3b2b978e9" containerID="6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b" exitCode=0
Jul 23 22:26:34.763678 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.763544    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"
Jul 23 22:26:34.763678 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.763585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j" event={"ID":"4d1607a7-4ae5-484e-993f-73f3b2b978e9","Type":"ContainerDied","Data":"6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b"}
Jul 23 22:26:34.763678 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.763633    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j" event={"ID":"4d1607a7-4ae5-484e-993f-73f3b2b978e9","Type":"ContainerDied","Data":"da8a604d3137601ca8012e7e88851561549cdb7ac59a90969174fa4e02602538"}
Jul 23 22:26:34.763678 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.763658    2579 scope.go:117] "RemoveContainer" containerID="6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b"
Jul 23 22:26:34.773039 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.773014    2579 scope.go:117] "RemoveContainer" containerID="6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b"
Jul 23 22:26:34.773341 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:26:34.773321    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b\": container with ID starting with 6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b not found: ID does not exist" containerID="6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b"
Jul 23 22:26:34.773398 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.773351    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b"} err="failed to get container status \"6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b\": rpc error: code = NotFound desc = could not find container \"6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b\": container with ID starting with 6f47ae38149377d1cb9ca1df6665d456c5ba9fce95cfa702e3297453c40bc07b not found: ID does not exist"
Jul 23 22:26:34.784188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.784158    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"]
Jul 23 22:26:34.787515 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:34.787488    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-82711a26-77889cb55f-nqs9j"]
Jul 23 22:26:35.111665 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:35.111629    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d1607a7-4ae5-484e-993f-73f3b2b978e9" path="/var/lib/kubelet/pods/4d1607a7-4ae5-484e-993f-73f3b2b978e9/volumes"
Jul 23 22:26:37.784784 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:37.784742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:26:37.885772 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:37.885730    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:26:38.822014 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:38.821978    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"]
Jul 23 22:26:38.823371 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:38.823348    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:26:38.825938 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:38.825915    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82711a26-gfntk"]
Jul 23 22:26:39.111405 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:39.111324    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="122be7a3-2a68-4ac5-9af6-360894a68a55" path="/var/lib/kubelet/pods/122be7a3-2a68-4ac5-9af6-360894a68a55/volumes"
Jul 23 22:26:41.126003 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:41.125965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:26:45.136723 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:45.136698    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:26:45.139156 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:45.139134    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:26:45.232809 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:45.232776    2579 scope.go:117] "RemoveContainer" containerID="282832f698c3aefcbeb3092d995fa533896fa2a7740c2b946aea1a9e2d1f374d"
Jul 23 22:26:46.111728 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:46.111693    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"]
Jul 23 22:26:46.156029 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:26:46.155992    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:01.133797 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.133719    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h" podUID="a1048f8a-d3c8-446e-98e7-b1dffa54eba4" containerName="main" containerID="cri-o://e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e" gracePeriod=45
Jul 23 22:27:01.387934 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.387844    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:27:01.506149 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.506109    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6mtj\" (UniqueName: \"kubernetes.io/projected/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-kube-api-access-z6mtj\") pod \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") "
Jul 23 22:27:01.506351 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.506174    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-home\") pod \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") "
Jul 23 22:27:01.506351 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.506192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-model-cache\") pod \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") "
Jul 23 22:27:01.506351 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.506209    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-dshm\") pod \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") "
Jul 23 22:27:01.506351 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.506232    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tmp-dir\") pod \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") "
Jul 23 22:27:01.506351 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.506266    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tls-certs\") pod \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\" (UID: \"a1048f8a-d3c8-446e-98e7-b1dffa54eba4\") "
Jul 23 22:27:01.506622 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.506502    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-home" (OuterVolumeSpecName: "home") pod "a1048f8a-d3c8-446e-98e7-b1dffa54eba4" (UID: "a1048f8a-d3c8-446e-98e7-b1dffa54eba4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:27:01.506622 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.506514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1048f8a-d3c8-446e-98e7-b1dffa54eba4" (UID: "a1048f8a-d3c8-446e-98e7-b1dffa54eba4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:27:01.506622 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.506519    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1048f8a-d3c8-446e-98e7-b1dffa54eba4" (UID: "a1048f8a-d3c8-446e-98e7-b1dffa54eba4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:27:01.508658 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.508635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-dshm" (OuterVolumeSpecName: "dshm") pod "a1048f8a-d3c8-446e-98e7-b1dffa54eba4" (UID: "a1048f8a-d3c8-446e-98e7-b1dffa54eba4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:27:01.508658 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.508650    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-kube-api-access-z6mtj" (OuterVolumeSpecName: "kube-api-access-z6mtj") pod "a1048f8a-d3c8-446e-98e7-b1dffa54eba4" (UID: "a1048f8a-d3c8-446e-98e7-b1dffa54eba4"). InnerVolumeSpecName "kube-api-access-z6mtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:27:01.508760 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.508709    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1048f8a-d3c8-446e-98e7-b1dffa54eba4" (UID: "a1048f8a-d3c8-446e-98e7-b1dffa54eba4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:27:01.606993 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.606947    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:01.606993 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.606985    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6mtj\" (UniqueName: \"kubernetes.io/projected/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-kube-api-access-z6mtj\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:01.606993 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.606997    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:01.607247 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.607009    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:01.607247 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.607020    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:01.607247 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.607030    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1048f8a-d3c8-446e-98e7-b1dffa54eba4-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:01.889729 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.889692    2579 generic.go:356] "Generic (PLEG): container finished" podID="a1048f8a-d3c8-446e-98e7-b1dffa54eba4" containerID="e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e" exitCode=0
Jul 23 22:27:01.889937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.889741    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"
Jul 23 22:27:01.889937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.889786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h" event={"ID":"a1048f8a-d3c8-446e-98e7-b1dffa54eba4","Type":"ContainerDied","Data":"e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e"}
Jul 23 22:27:01.889937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.889837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h" event={"ID":"a1048f8a-d3c8-446e-98e7-b1dffa54eba4","Type":"ContainerDied","Data":"8564cdaf28d823298a3a26abf64664ee09e1755320e488d728cf69be6ad5e11d"}
Jul 23 22:27:01.889937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.889889    2579 scope.go:117] "RemoveContainer" containerID="e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e"
Jul 23 22:27:01.899323 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.899306    2579 scope.go:117] "RemoveContainer" containerID="e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e"
Jul 23 22:27:01.899624 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:27:01.899603    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e\": container with ID starting with e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e not found: ID does not exist" containerID="e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e"
Jul 23 22:27:01.899684 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.899635    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e"} err="failed to get container status \"e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e\": rpc error: code = NotFound desc = could not find container \"e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e\": container with ID starting with e7e2fdcc835140b676f7478a30ca47d8456a7e14a085dec7cdcfe7f5b4d6489e not found: ID does not exist"
Jul 23 22:27:01.910123 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.910087    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"]
Jul 23 22:27:01.911817 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:01.911792    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-82711a26/e2e-lifecycle-model-82711a26-kserve-6b7bf54499-7629h"]
Jul 23 22:27:03.112016 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:03.111985    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1048f8a-d3c8-446e-98e7-b1dffa54eba4" path="/var/lib/kubelet/pods/a1048f8a-d3c8-446e-98e7-b1dffa54eba4/volumes"
Jul 23 22:27:23.453462 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:23.453419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:23.727916 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:23.727804    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:30.170144 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:30.170102    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:30.272794 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:30.272762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:30.451757 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:30.451728    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"]
Jul 23 22:27:30.451999 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:30.451957    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy" containerID="cri-o://e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75" gracePeriod=30
Jul 23 22:27:30.477780 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:30.477749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:30.723128 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:30.723039    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:31.796232 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:31.796196    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:31.896426 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:31.896394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:31.996266 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:31.996230    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:32.782171 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:32.782133    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:32.881991 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:32.881944    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:32.981647 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:32.981616    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:34.529599 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:34.529565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:35.714095 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.714026    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:27:35.807635 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.807589    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-socket\") pod \"e2dbe01d-81fb-4f03-be67-3c00938fd728\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") "
Jul 23 22:27:35.807635 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.807644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-data\") pod \"e2dbe01d-81fb-4f03-be67-3c00938fd728\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") "
Jul 23 22:27:35.807908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.807703    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-certs\") pod \"e2dbe01d-81fb-4f03-be67-3c00938fd728\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") "
Jul 23 22:27:35.807908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.807723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2dbe01d-81fb-4f03-be67-3c00938fd728-istiod-ca-cert\") pod \"e2dbe01d-81fb-4f03-be67-3c00938fd728\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") "
Jul 23 22:27:35.807908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.807755    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-token\") pod \"e2dbe01d-81fb-4f03-be67-3c00938fd728\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") "
Jul 23 22:27:35.807908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.807785    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-podinfo\") pod \"e2dbe01d-81fb-4f03-be67-3c00938fd728\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") "
Jul 23 22:27:35.807908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.807823    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmvs7\" (UniqueName: \"kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-kube-api-access-xmvs7\") pod \"e2dbe01d-81fb-4f03-be67-3c00938fd728\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") "
Jul 23 22:27:35.807908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.807890    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-envoy\") pod \"e2dbe01d-81fb-4f03-be67-3c00938fd728\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") "
Jul 23 22:27:35.808228 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.807908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2dbe01d-81fb-4f03-be67-3c00938fd728" (UID: "e2dbe01d-81fb-4f03-be67-3c00938fd728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:27:35.808228 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.807923    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-credential-socket\") pod \"e2dbe01d-81fb-4f03-be67-3c00938fd728\" (UID: \"e2dbe01d-81fb-4f03-be67-3c00938fd728\") "
Jul 23 22:27:35.808228 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.808060    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2dbe01d-81fb-4f03-be67-3c00938fd728" (UID: "e2dbe01d-81fb-4f03-be67-3c00938fd728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:27:35.808228 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.808178    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2dbe01d-81fb-4f03-be67-3c00938fd728-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2dbe01d-81fb-4f03-be67-3c00938fd728" (UID: "e2dbe01d-81fb-4f03-be67-3c00938fd728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:27:35.808471 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.808240    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:35.808471 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.808260    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-workload-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:35.808471 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.808460    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2dbe01d-81fb-4f03-be67-3c00938fd728" (UID: "e2dbe01d-81fb-4f03-be67-3c00938fd728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:27:35.808960 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.808933    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2dbe01d-81fb-4f03-be67-3c00938fd728" (UID: "e2dbe01d-81fb-4f03-be67-3c00938fd728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:27:35.810246 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.810205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-kube-api-access-xmvs7" (OuterVolumeSpecName: "kube-api-access-xmvs7") pod "e2dbe01d-81fb-4f03-be67-3c00938fd728" (UID: "e2dbe01d-81fb-4f03-be67-3c00938fd728"). InnerVolumeSpecName "kube-api-access-xmvs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:27:35.810463 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.810271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2dbe01d-81fb-4f03-be67-3c00938fd728" (UID: "e2dbe01d-81fb-4f03-be67-3c00938fd728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:27:35.810607 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.810590    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2dbe01d-81fb-4f03-be67-3c00938fd728" (UID: "e2dbe01d-81fb-4f03-be67-3c00938fd728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 22:27:35.810684 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.810659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2dbe01d-81fb-4f03-be67-3c00938fd728" (UID: "e2dbe01d-81fb-4f03-be67-3c00938fd728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:27:35.909120 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.909073    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmvs7\" (UniqueName: \"kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-kube-api-access-xmvs7\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:35.909120 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.909113    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-envoy\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:35.909120 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.909127    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-credential-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:35.909371 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.909140    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-data\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:35.909371 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.909151    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2dbe01d-81fb-4f03-be67-3c00938fd728-istiod-ca-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:35.909371 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.909162    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-token\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:35.909371 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:35.909173    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2dbe01d-81fb-4f03-be67-3c00938fd728-istio-podinfo\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:27:36.048997 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.048906    2579 generic.go:356] "Generic (PLEG): container finished" podID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerID="e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75" exitCode=0
Jul 23 22:27:36.048997 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.048955    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"
Jul 23 22:27:36.049220 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.048956    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" event={"ID":"e2dbe01d-81fb-4f03-be67-3c00938fd728","Type":"ContainerDied","Data":"e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75"}
Jul 23 22:27:36.049220 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.049121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698" event={"ID":"e2dbe01d-81fb-4f03-be67-3c00938fd728","Type":"ContainerDied","Data":"2901928f7cee0665ef2f0056ad2d6e6724cf0c4488e39f75be0b3e5862148e27"}
Jul 23 22:27:36.049220 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.049160    2579 scope.go:117] "RemoveContainer" containerID="e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75"
Jul 23 22:27:36.058221 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.058203    2579 scope.go:117] "RemoveContainer" containerID="7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b"
Jul 23 22:27:36.066504 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.066480    2579 scope.go:117] "RemoveContainer" containerID="e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75"
Jul 23 22:27:36.066847 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:27:36.066828    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75\": container with ID starting with e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75 not found: ID does not exist" containerID="e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75"
Jul 23 22:27:36.066905 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.066874    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75"} err="failed to get container status \"e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75\": rpc error: code = NotFound desc = could not find container \"e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75\": container with ID starting with e81199d188c248d573f19c4ed4f16dc1f8371bf272d2be329b4ea84ac3c8cd75 not found: ID does not exist"
Jul 23 22:27:36.066905 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.066896    2579 scope.go:117] "RemoveContainer" containerID="7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b"
Jul 23 22:27:36.067128 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:27:36.067113    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b\": container with ID starting with 7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b not found: ID does not exist" containerID="7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b"
Jul 23 22:27:36.067183 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.067132    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b"} err="failed to get container status \"7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b\": rpc error: code = NotFound desc = could not find container \"7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b\": container with ID starting with 7312d08ff478271e9ca11c96ac306cd36dcda99b63b76caf6c6a6a476a16262b not found: ID does not exist"
Jul 23 22:27:36.072612 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.072588    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"]
Jul 23 22:27:36.076692 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:36.076665    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-82711a26-openshift-default-79f75d87bc-cq698"]
Jul 23 22:27:37.111537 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:37.111504    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" path="/var/lib/kubelet/pods/e2dbe01d-81fb-4f03-be67-3c00938fd728/volumes"
Jul 23 22:27:41.410524 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:41.410492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:41.508855 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:41.508820    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:42.133357 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:42.133317    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:45.256454 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:45.256420    2579 scope.go:117] "RemoveContainer" containerID="d1aabb4d2c88d6a447a9755fa32ec6645e05cbdfd35b03cf7fb36e8acea11919"
Jul 23 22:27:48.643388 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:48.643352    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:48.754420 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:48.754382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:48.944168 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:48.944085    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:59.404290 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:59.404253    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:27:59.761482 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:27:59.761449    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:20.796551 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:20.796517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:27.800391 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:27.800356    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:27.862237 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:27.862205    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:28.176266 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.176169    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:28.486365 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486333    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"]
Jul 23 22:28:28.486640 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486628    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d1607a7-4ae5-484e-993f-73f3b2b978e9" containerName="maas-api"
Jul 23 22:28:28.486685 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486641    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d1607a7-4ae5-484e-993f-73f3b2b978e9" containerName="maas-api"
Jul 23 22:28:28.486685 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486659    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1048f8a-d3c8-446e-98e7-b1dffa54eba4" containerName="main"
Jul 23 22:28:28.486685 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486665    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1048f8a-d3c8-446e-98e7-b1dffa54eba4" containerName="main"
Jul 23 22:28:28.486685 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486672    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy"
Jul 23 22:28:28.486685 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486678    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy"
Jul 23 22:28:28.486881 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486690    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="122be7a3-2a68-4ac5-9af6-360894a68a55" containerName="revoke-keys"
Jul 23 22:28:28.486881 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486695    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="122be7a3-2a68-4ac5-9af6-360894a68a55" containerName="revoke-keys"
Jul 23 22:28:28.486881 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486739    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy"
Jul 23 22:28:28.486881 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486749    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1048f8a-d3c8-446e-98e7-b1dffa54eba4" containerName="main"
Jul 23 22:28:28.486881 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486757    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="122be7a3-2a68-4ac5-9af6-360894a68a55" containerName="revoke-keys"
Jul 23 22:28:28.486881 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.486762    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d1607a7-4ae5-484e-993f-73f3b2b978e9" containerName="maas-api"
Jul 23 22:28:28.489709 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.489691    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.492266 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.492238    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8ba6ab-openshift-default-dockercfg-ngldp\""
Jul 23 22:28:28.492382 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.492238    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 22:28:28.501028 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.501001    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"]
Jul 23 22:28:28.513653 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.513624    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:28.663293 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.663250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-socket\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.663293 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.663292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732355aa-5655-466b-b974-496329d2582c-istio-podinfo\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.663519 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.663347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-envoy\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.663519 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.663385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbt4\" (UniqueName: \"kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-kube-api-access-tgbt4\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.663519 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.663441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-certs\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.663519 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.663479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-data\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.663519 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.663499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-istio-token\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.663683 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.663524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-credential-socket\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.663683 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.663543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732355aa-5655-466b-b974-496329d2582c-istiod-ca-cert\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.694702 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.694661    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:28.764756 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.764671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-certs\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.764756 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.764709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-data\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.764756 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.764730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-istio-token\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.764756 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.764754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-credential-socket\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.765110 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.764770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732355aa-5655-466b-b974-496329d2582c-istiod-ca-cert\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.765110 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.764795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-socket\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.765110 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.764920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732355aa-5655-466b-b974-496329d2582c-istio-podinfo\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.765110 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.764998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-envoy\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.765110 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.765066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbt4\" (UniqueName: \"kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-kube-api-access-tgbt4\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.765362 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.765187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-data\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.765362 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.765239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-certs\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.765496 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.765390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-credential-socket\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.765538 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.765498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-socket\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.765576 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.765544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732355aa-5655-466b-b974-496329d2582c-istiod-ca-cert\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.767476 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.767456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-envoy\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.767669 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.767648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732355aa-5655-466b-b974-496329d2582c-istio-podinfo\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.772413 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.772387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-istio-token\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.772506 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.772455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbt4\" (UniqueName: \"kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-kube-api-access-tgbt4\") pod \"e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") " pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.799548 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.799505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:28.938464 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.938295    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"]
Jul 23 22:28:28.941253 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:28:28.941226    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732355aa_5655_466b_b974_496329d2582c.slice/crio-4e7bcdcb0fca2961143cfa700c0200128e820a9f62b860a5caf70a14cdd94fcb WatchSource:0}: Error finding container 4e7bcdcb0fca2961143cfa700c0200128e820a9f62b860a5caf70a14cdd94fcb: Status 404 returned error can't find the container with id 4e7bcdcb0fca2961143cfa700c0200128e820a9f62b860a5caf70a14cdd94fcb
Jul 23 22:28:28.943588 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.943550    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:28:28.943656 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.943645    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:28:28.943703 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:28.943688    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:28:29.286176 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:29.286139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" event={"ID":"732355aa-5655-466b-b974-496329d2582c","Type":"ContainerStarted","Data":"50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc"}
Jul 23 22:28:29.286356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:29.286193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" event={"ID":"732355aa-5655-466b-b974-496329d2582c","Type":"ContainerStarted","Data":"4e7bcdcb0fca2961143cfa700c0200128e820a9f62b860a5caf70a14cdd94fcb"}
Jul 23 22:28:29.307179 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:29.307121    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" podStartSLOduration=1.3071033 podStartE2EDuration="1.3071033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:28:29.305730566 +0000 UTC m=+2204.696123182" watchObservedRunningTime="2026-07-23 22:28:29.3071033 +0000 UTC m=+2204.697495916"
Jul 23 22:28:29.799898 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:29.799839    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:29.804979 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:29.804941    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:30.290029 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:30.289995    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:30.290994 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:30.290969    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:31.310734 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.310693    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:31.402960 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.402930    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:31.498082 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.498044    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"]
Jul 23 22:28:31.498353 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.498341    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy"
Jul 23 22:28:31.498398 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.498355    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy"
Jul 23 22:28:31.498433 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.498410    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2dbe01d-81fb-4f03-be67-3c00938fd728" containerName="istio-proxy"
Jul 23 22:28:31.501389 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.501371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:31.503841 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.503819    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8ba6ab-serving-cert\""
Jul 23 22:28:31.514404 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.514378    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"]
Jul 23 22:28:31.534291 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.534256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:31.693532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.693434    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd4273-173d-4304-8f41-37b9f93ef30e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:31.693532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.693493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbrv\" (UniqueName: \"kubernetes.io/projected/12cd4273-173d-4304-8f41-37b9f93ef30e-kube-api-access-drbrv\") pod \"maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:31.693755 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.693556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd4273-173d-4304-8f41-37b9f93ef30e-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:31.794968 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.794927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbrv\" (UniqueName: \"kubernetes.io/projected/12cd4273-173d-4304-8f41-37b9f93ef30e-kube-api-access-drbrv\") pod \"maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:31.794968 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.794968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd4273-173d-4304-8f41-37b9f93ef30e-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:31.795243 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.795029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd4273-173d-4304-8f41-37b9f93ef30e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:31.795243 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:28:31.795143    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-serving-cert: secret "maas-api-e2e-inf-a-8ba6ab-serving-cert" not found
Jul 23 22:28:31.795243 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:28:31.795204    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12cd4273-173d-4304-8f41-37b9f93ef30e-maas-api-tls podName:12cd4273-173d-4304-8f41-37b9f93ef30e nodeName:}" failed. No retries permitted until 2026-07-23 22:28:32.295189397 +0000 UTC m=+2207.685581991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12cd4273-173d-4304-8f41-37b9f93ef30e-maas-api-tls") pod "maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45" (UID: "12cd4273-173d-4304-8f41-37b9f93ef30e") : secret "maas-api-e2e-inf-a-8ba6ab-serving-cert" not found
Jul 23 22:28:31.795582 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.795563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd4273-173d-4304-8f41-37b9f93ef30e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:31.806282 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.806259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbrv\" (UniqueName: \"kubernetes.io/projected/12cd4273-173d-4304-8f41-37b9f93ef30e-kube-api-access-drbrv\") pod \"maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:31.999407 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:31.999362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:32.117222 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:32.117182    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:32.218918 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:32.218880    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:32.300090 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:32.299989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd4273-173d-4304-8f41-37b9f93ef30e-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:32.302644 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:32.302620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd4273-173d-4304-8f41-37b9f93ef30e-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:32.411802 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:32.411748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:32.554425 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:32.554398    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"]
Jul 23 22:28:32.556251 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:28:32.556219    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cd4273_173d_4304_8f41_37b9f93ef30e.slice/crio-1d9756d09eaa5dcda899ddf101bae3c756166429c88c9691749ae97326e0111b WatchSource:0}: Error finding container 1d9756d09eaa5dcda899ddf101bae3c756166429c88c9691749ae97326e0111b: Status 404 returned error can't find the container with id 1d9756d09eaa5dcda899ddf101bae3c756166429c88c9691749ae97326e0111b
Jul 23 22:28:32.557756 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:32.557733    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 22:28:33.302712 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:33.302681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45" event={"ID":"12cd4273-173d-4304-8f41-37b9f93ef30e","Type":"ContainerStarted","Data":"6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099"}
Jul 23 22:28:33.302969 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:33.302738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45" event={"ID":"12cd4273-173d-4304-8f41-37b9f93ef30e","Type":"ContainerStarted","Data":"1d9756d09eaa5dcda899ddf101bae3c756166429c88c9691749ae97326e0111b"}
Jul 23 22:28:33.302969 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:33.302762    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:33.321070 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:33.321024    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45" podStartSLOduration=1.7426338540000001 podStartE2EDuration="2.321007075s" totalImagesPullingTime="578.373221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:28:33.319951828 +0000 UTC m=+2208.710344443" watchObservedRunningTime="2026-07-23 22:28:33.321007075 +0000 UTC m=+2208.711399691"
Jul 23 22:28:35.871694 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:35.871664    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:36.230528 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:36.230491    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:36.551354 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:36.551238    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:36.682886 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:36.682833    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:39.313375 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:39.313344    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:28:39.772523 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:39.772479    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:39.854342 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:39.854314    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"]
Jul 23 22:28:39.854509 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:39.854411    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:39.857760 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:39.857736    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:39.860202 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:39.860176    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8a3e27-serving-cert\""
Jul 23 22:28:39.868824 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:39.868802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"]
Jul 23 22:28:39.934210 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:39.934177    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:39.957765 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:39.957720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:39.957986 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:39.957789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:39.957986 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:39.957827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvvpv\" (UniqueName: \"kubernetes.io/projected/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-kube-api-access-fvvpv\") pod \"maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:40.033304 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:40.033223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:40.059087 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:40.059046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:40.059289 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:40.059095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvpv\" (UniqueName: \"kubernetes.io/projected/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-kube-api-access-fvvpv\") pod \"maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:40.059289 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:40.059142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:40.059799 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:40.059775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:40.061742 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:40.061715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-maas-api-tls\") pod \"maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:40.068006 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:40.067958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvvpv\" (UniqueName: \"kubernetes.io/projected/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-kube-api-access-fvvpv\") pod \"maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:40.169813 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:40.169756    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:40.345145 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:40.345064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"]
Jul 23 22:28:40.348240 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:28:40.348204    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b14797_c7d7_47a7_96f3_6c0a336dd0e0.slice/crio-5a1beb66fb0b0c1292feeeed5835e0462700cb5239c3be9612aecb95c114cc21 WatchSource:0}: Error finding container 5a1beb66fb0b0c1292feeeed5835e0462700cb5239c3be9612aecb95c114cc21: Status 404 returned error can't find the container with id 5a1beb66fb0b0c1292feeeed5835e0462700cb5239c3be9612aecb95c114cc21
Jul 23 22:28:40.447581 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:40.447551    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:40.546270 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:40.546235    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:41.339842 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:41.339804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc" event={"ID":"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0","Type":"ContainerStarted","Data":"69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669"}
Jul 23 22:28:41.340081 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:41.339887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc" event={"ID":"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0","Type":"ContainerStarted","Data":"5a1beb66fb0b0c1292feeeed5835e0462700cb5239c3be9612aecb95c114cc21"}
Jul 23 22:28:41.340081 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:41.339933    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:41.357822 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:41.357770    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc" podStartSLOduration=2.025748146 podStartE2EDuration="2.357753001s" totalImagesPullingTime="332.004855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:28:41.356342967 +0000 UTC m=+2216.746735583" watchObservedRunningTime="2026-07-23 22:28:41.357753001 +0000 UTC m=+2216.748145670"
Jul 23 22:28:44.290127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:44.290088    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:45.281210 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:45.281127    2579 scope.go:117] "RemoveContainer" containerID="1f6922666c5e28f8e3e3fedebf2daae66af4067e2b4e2a83afc653e4cc8b8b2e"
Jul 23 22:28:45.289325 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:45.289302    2579 scope.go:117] "RemoveContainer" containerID="3adb8ad25c536e4101e403ccd26ee8cf9245926018b0c8a2b46ce1805b5e57de"
Jul 23 22:28:45.291216 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:45.291193    2579 patch_prober.go:28] interesting pod/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 23 22:28:45.291530 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:45.291247    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 23 22:28:45.360195 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:45.360104    2579 generic.go:356] "Generic (PLEG): container finished" podID="732355aa-5655-466b-b974-496329d2582c" containerID="50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc" exitCode=0
Jul 23 22:28:45.360195 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:45.360146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" event={"ID":"732355aa-5655-466b-b974-496329d2582c","Type":"ContainerDied","Data":"50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc"}
Jul 23 22:28:45.360619 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:45.360599    2579 scope.go:117] "RemoveContainer" containerID="50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc"
Jul 23 22:28:45.361571 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:45.361537    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:28:45.361704 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:45.361609    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:28:45.361704 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:45.361642    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:28:46.381218 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:46.381178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" event={"ID":"732355aa-5655-466b-b974-496329d2582c","Type":"ContainerStarted","Data":"9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231"}
Jul 23 22:28:46.800458 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:46.800414    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:46.800649 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:46.800606    2579 patch_prober.go:28] interesting pod/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 23 22:28:46.800700 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:46.800657    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 23 22:28:47.349207 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:47.349178    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:28:47.800193 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:47.800153    2579 patch_prober.go:28] interesting pod/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 23 22:28:47.800604 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:47.800218    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 23 22:28:48.800237 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:48.800196    2579 patch_prober.go:28] interesting pod/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 23 22:28:48.800650 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:48.800259    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 23 22:28:49.800270 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:49.800233    2579 patch_prober.go:28] interesting pod/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Jul 23 22:28:49.800686 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:49.800294    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Jul 23 22:28:50.804118 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:50.804085    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:50.804513 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:50.804360    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:50.805133 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:50.805117    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:28:53.940133 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:53.940093    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:54.612834 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.612797    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"]
Jul 23 22:28:54.615399 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.615376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.618448 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.618426    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8ba6ab\"/\"openshift-service-ca.crt\""
Jul 23 22:28:54.618640 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.618425    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8ba6ab\"/\"test-model-8ba6ab-kserve-self-signed-certs\""
Jul 23 22:28:54.618765 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.618440    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8ba6ab\"/\"kube-root-ca.crt\""
Jul 23 22:28:54.618765 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.618446    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8ba6ab\"/\"default-dockercfg-hbjk9\""
Jul 23 22:28:54.624557 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.624534    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"]
Jul 23 22:28:54.700217 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.700178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tls-certs\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.700217 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.700218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjbt5\" (UniqueName: \"kubernetes.io/projected/e9a5f046-6b50-42d1-8d7f-7b13166614ae-kube-api-access-wjbt5\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.700442 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.700243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-home\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.700442 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.700276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-dshm\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.700442 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.700345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-model-cache\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.700442 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.700425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tmp-dir\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.801814 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.801767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-home\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.801814 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.801818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-dshm\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.802091 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.801851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-model-cache\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.802091 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.801937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tmp-dir\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.802091 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.801970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tls-certs\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.802091 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.801987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjbt5\" (UniqueName: \"kubernetes.io/projected/e9a5f046-6b50-42d1-8d7f-7b13166614ae-kube-api-access-wjbt5\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.802302 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.802246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-home\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.802302 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.802287    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tmp-dir\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.802376 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.802349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-model-cache\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.804309 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.804277    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-dshm\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.804541 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.804526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tls-certs\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.810550 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.810521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjbt5\" (UniqueName: \"kubernetes.io/projected/e9a5f046-6b50-42d1-8d7f-7b13166614ae-kube-api-access-wjbt5\") pod \"test-model-8ba6ab-kserve-6b654c87cc-qmvkz\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") " pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:54.926264 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:54.926162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:28:55.067402 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:55.067372    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"]
Jul 23 22:28:55.069217 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:28:55.069192    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9a5f046_6b50_42d1_8d7f_7b13166614ae.slice/crio-e3d7cdf8951426fef3fe46638f07de01657ffd0946a0b1d28f6d3f3a39e0fd53 WatchSource:0}: Error finding container e3d7cdf8951426fef3fe46638f07de01657ffd0946a0b1d28f6d3f3a39e0fd53: Status 404 returned error can't find the container with id e3d7cdf8951426fef3fe46638f07de01657ffd0946a0b1d28f6d3f3a39e0fd53
Jul 23 22:28:55.118334 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:55.118305    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:28:55.442226 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:55.442190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz" event={"ID":"e9a5f046-6b50-42d1-8d7f-7b13166614ae","Type":"ContainerStarted","Data":"26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0"}
Jul 23 22:28:55.442407 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:55.442238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz" event={"ID":"e9a5f046-6b50-42d1-8d7f-7b13166614ae","Type":"ContainerStarted","Data":"e3d7cdf8951426fef3fe46638f07de01657ffd0946a0b1d28f6d3f3a39e0fd53"}
Jul 23 22:28:55.459847 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:28:55.459791    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz" podStartSLOduration=1.459776072 podStartE2EDuration="1.459776072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:28:55.458022799 +0000 UTC m=+2230.848415415" watchObservedRunningTime="2026-07-23 22:28:55.459776072 +0000 UTC m=+2230.850168973"
Jul 23 22:29:01.347528 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:01.347493    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:29:01.447908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:01.447854    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:29:04.927210 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:04.927169    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:29:04.927210 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:04.927217    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:29:04.939882 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:04.939837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:29:05.499909 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:05.499841    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:29:13.423357 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:13.423319    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:29:16.374788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:16.374749    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:29:16.429689 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:16.429656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:29:18.701402 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.701367    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"]
Jul 23 22:29:18.704023 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.704005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.706220 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.706199    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8a3e27\"/\"default-dockercfg-tv8h5\""
Jul 23 22:29:18.706415 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.706251    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8a3e27\"/\"openshift-service-ca.crt\""
Jul 23 22:29:18.707188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.707168    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8a3e27\"/\"kube-root-ca.crt\""
Jul 23 22:29:18.707307 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.707226    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8a3e27\"/\"test-model-8a3e27-kserve-self-signed-certs\""
Jul 23 22:29:18.711806 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.711772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-dshm\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.711947 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.711808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-model-cache\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.711947 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.711834    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95c98fe1-b72e-40f5-b35c-23f4b669690b-tls-certs\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.711947 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.711902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-home\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.712128 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.711971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-tmp-dir\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.712128 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.712005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86722\" (UniqueName: \"kubernetes.io/projected/95c98fe1-b72e-40f5-b35c-23f4b669690b-kube-api-access-86722\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.719117 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.719086    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"]
Jul 23 22:29:18.736946 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.736909    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:29:18.813290 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.813254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86722\" (UniqueName: \"kubernetes.io/projected/95c98fe1-b72e-40f5-b35c-23f4b669690b-kube-api-access-86722\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.813475 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.813348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-dshm\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.813475 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.813368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-model-cache\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.813475 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.813383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95c98fe1-b72e-40f5-b35c-23f4b669690b-tls-certs\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.813475 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.813419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-home\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.813475 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.813463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-tmp-dir\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.813883 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.813835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-model-cache\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.813883 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.813877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-tmp-dir\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.814008 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.813903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-home\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.815932 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.815915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-dshm\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.816133 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.816117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95c98fe1-b72e-40f5-b35c-23f4b669690b-tls-certs\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:18.821957 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:18.821930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86722\" (UniqueName: \"kubernetes.io/projected/95c98fe1-b72e-40f5-b35c-23f4b669690b-kube-api-access-86722\") pod \"test-model-8a3e27-kserve-6f8578b95c-czp44\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") " pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:19.014063 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:19.013963    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:19.154412 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:19.154368    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"]
Jul 23 22:29:19.156033 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:29:19.156004    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95c98fe1_b72e_40f5_b35c_23f4b669690b.slice/crio-9f4bfca2c7b766249c452987a28a3d807c5398939c42ae52e31256d881a8ab90 WatchSource:0}: Error finding container 9f4bfca2c7b766249c452987a28a3d807c5398939c42ae52e31256d881a8ab90: Status 404 returned error can't find the container with id 9f4bfca2c7b766249c452987a28a3d807c5398939c42ae52e31256d881a8ab90
Jul 23 22:29:19.550147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:19.550113    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44" event={"ID":"95c98fe1-b72e-40f5-b35c-23f4b669690b","Type":"ContainerStarted","Data":"f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346"}
Jul 23 22:29:19.550147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:19.550152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44" event={"ID":"95c98fe1-b72e-40f5-b35c-23f4b669690b","Type":"ContainerStarted","Data":"9f4bfca2c7b766249c452987a28a3d807c5398939c42ae52e31256d881a8ab90"}
Jul 23 22:29:19.568954 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:19.568899    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44" podStartSLOduration=1.568881755 podStartE2EDuration="1.568881755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:29:19.568535687 +0000 UTC m=+2254.958928315" watchObservedRunningTime="2026-07-23 22:29:19.568881755 +0000 UTC m=+2254.959274367"
Jul 23 22:29:29.014289 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:29.014247    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:29.014775 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:29.014379    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:29.027086 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:29.027051    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:29.605658 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:29.605623    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:29:33.943618 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:33.943565    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:29:58.952624 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:58.952580    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"]
Jul 23 22:29:58.993392 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:58.993355    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:29:59.890803 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:59.890772    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"]
Jul 23 22:29:59.911069 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:29:59.911036    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:00.765036 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:00.765003    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:00.867646 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:00.867610    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:10.465976 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:10.465933    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"]
Jul 23 22:30:10.489710 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:10.489657    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:10.774494 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:10.774386    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45" podUID="12cd4273-173d-4304-8f41-37b9f93ef30e" containerName="maas-api" containerID="cri-o://6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099" gracePeriod=30
Jul 23 22:30:11.023245 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.023216    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:30:11.089472 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.089380    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drbrv\" (UniqueName: \"kubernetes.io/projected/12cd4273-173d-4304-8f41-37b9f93ef30e-kube-api-access-drbrv\") pod \"12cd4273-173d-4304-8f41-37b9f93ef30e\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") "
Jul 23 22:30:11.089472 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.089441    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd4273-173d-4304-8f41-37b9f93ef30e-openshift-service-ca\") pod \"12cd4273-173d-4304-8f41-37b9f93ef30e\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") "
Jul 23 22:30:11.089657 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.089545    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd4273-173d-4304-8f41-37b9f93ef30e-maas-api-tls\") pod \"12cd4273-173d-4304-8f41-37b9f93ef30e\" (UID: \"12cd4273-173d-4304-8f41-37b9f93ef30e\") "
Jul 23 22:30:11.089824 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.089800    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cd4273-173d-4304-8f41-37b9f93ef30e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12cd4273-173d-4304-8f41-37b9f93ef30e" (UID: "12cd4273-173d-4304-8f41-37b9f93ef30e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:30:11.091733 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.091695    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12cd4273-173d-4304-8f41-37b9f93ef30e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12cd4273-173d-4304-8f41-37b9f93ef30e" (UID: "12cd4273-173d-4304-8f41-37b9f93ef30e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:30:11.091843 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.091731    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12cd4273-173d-4304-8f41-37b9f93ef30e-kube-api-access-drbrv" (OuterVolumeSpecName: "kube-api-access-drbrv") pod "12cd4273-173d-4304-8f41-37b9f93ef30e" (UID: "12cd4273-173d-4304-8f41-37b9f93ef30e"). InnerVolumeSpecName "kube-api-access-drbrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:30:11.190357 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.190316    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drbrv\" (UniqueName: \"kubernetes.io/projected/12cd4273-173d-4304-8f41-37b9f93ef30e-kube-api-access-drbrv\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:11.190357 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.190351    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cd4273-173d-4304-8f41-37b9f93ef30e-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:11.190357 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.190361    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12cd4273-173d-4304-8f41-37b9f93ef30e-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:11.779621 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.779583    2579 generic.go:356] "Generic (PLEG): container finished" podID="12cd4273-173d-4304-8f41-37b9f93ef30e" containerID="6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099" exitCode=0
Jul 23 22:30:11.780127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.779630    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"
Jul 23 22:30:11.780127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.779669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45" event={"ID":"12cd4273-173d-4304-8f41-37b9f93ef30e","Type":"ContainerDied","Data":"6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099"}
Jul 23 22:30:11.780127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.779730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45" event={"ID":"12cd4273-173d-4304-8f41-37b9f93ef30e","Type":"ContainerDied","Data":"1d9756d09eaa5dcda899ddf101bae3c756166429c88c9691749ae97326e0111b"}
Jul 23 22:30:11.780127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.779762    2579 scope.go:117] "RemoveContainer" containerID="6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099"
Jul 23 22:30:11.788184 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.788162    2579 scope.go:117] "RemoveContainer" containerID="6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099"
Jul 23 22:30:11.788452 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:30:11.788433    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099\": container with ID starting with 6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099 not found: ID does not exist" containerID="6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099"
Jul 23 22:30:11.788515 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.788464    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099"} err="failed to get container status \"6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099\": rpc error: code = NotFound desc = could not find container \"6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099\": container with ID starting with 6c579d94c17a27c6acf8d8dbc21ba5e100072cdc9bc900b496f37fba05efe099 not found: ID does not exist"
Jul 23 22:30:11.795345 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.795315    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"]
Jul 23 22:30:11.798741 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:11.798715    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ba6ab-7b9dd4b559-z4s45"]
Jul 23 22:30:13.111585 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:13.111549    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12cd4273-173d-4304-8f41-37b9f93ef30e" path="/var/lib/kubelet/pods/12cd4273-173d-4304-8f41-37b9f93ef30e/volumes"
Jul 23 22:30:13.973372 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:13.973328    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz" podUID="e9a5f046-6b50-42d1-8d7f-7b13166614ae" containerName="main" containerID="cri-o://26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0" gracePeriod=45
Jul 23 22:30:14.229297 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.229226    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:30:14.316389 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.316358    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tmp-dir\") pod \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") "
Jul 23 22:30:14.316555 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.316404    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-home\") pod \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") "
Jul 23 22:30:14.316555 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.316421    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-dshm\") pod \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") "
Jul 23 22:30:14.316555 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.316451    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tls-certs\") pod \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") "
Jul 23 22:30:14.316555 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.316481    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-model-cache\") pod \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") "
Jul 23 22:30:14.316761 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.316560    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjbt5\" (UniqueName: \"kubernetes.io/projected/e9a5f046-6b50-42d1-8d7f-7b13166614ae-kube-api-access-wjbt5\") pod \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\" (UID: \"e9a5f046-6b50-42d1-8d7f-7b13166614ae\") "
Jul 23 22:30:14.316761 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.316658    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9a5f046-6b50-42d1-8d7f-7b13166614ae" (UID: "e9a5f046-6b50-42d1-8d7f-7b13166614ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:14.316761 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.316668    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-home" (OuterVolumeSpecName: "home") pod "e9a5f046-6b50-42d1-8d7f-7b13166614ae" (UID: "e9a5f046-6b50-42d1-8d7f-7b13166614ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:14.316959 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.316886    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9a5f046-6b50-42d1-8d7f-7b13166614ae" (UID: "e9a5f046-6b50-42d1-8d7f-7b13166614ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:14.316959 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.316948    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:14.317081 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.316970    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:14.318836 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.318805    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-dshm" (OuterVolumeSpecName: "dshm") pod "e9a5f046-6b50-42d1-8d7f-7b13166614ae" (UID: "e9a5f046-6b50-42d1-8d7f-7b13166614ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:14.319017 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.318885    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a5f046-6b50-42d1-8d7f-7b13166614ae-kube-api-access-wjbt5" (OuterVolumeSpecName: "kube-api-access-wjbt5") pod "e9a5f046-6b50-42d1-8d7f-7b13166614ae" (UID: "e9a5f046-6b50-42d1-8d7f-7b13166614ae"). InnerVolumeSpecName "kube-api-access-wjbt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:30:14.319017 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.318984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9a5f046-6b50-42d1-8d7f-7b13166614ae" (UID: "e9a5f046-6b50-42d1-8d7f-7b13166614ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:30:14.417803 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.417752    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjbt5\" (UniqueName: \"kubernetes.io/projected/e9a5f046-6b50-42d1-8d7f-7b13166614ae-kube-api-access-wjbt5\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:14.417803 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.417793    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:14.417803 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.417807    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a5f046-6b50-42d1-8d7f-7b13166614ae-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:14.418095 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.417821    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a5f046-6b50-42d1-8d7f-7b13166614ae-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:14.794969 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.794936    2579 generic.go:356] "Generic (PLEG): container finished" podID="e9a5f046-6b50-42d1-8d7f-7b13166614ae" containerID="26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0" exitCode=0
Jul 23 22:30:14.795147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.794985    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"
Jul 23 22:30:14.795147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.794989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz" event={"ID":"e9a5f046-6b50-42d1-8d7f-7b13166614ae","Type":"ContainerDied","Data":"26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0"}
Jul 23 22:30:14.795147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.795109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz" event={"ID":"e9a5f046-6b50-42d1-8d7f-7b13166614ae","Type":"ContainerDied","Data":"e3d7cdf8951426fef3fe46638f07de01657ffd0946a0b1d28f6d3f3a39e0fd53"}
Jul 23 22:30:14.795147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.795135    2579 scope.go:117] "RemoveContainer" containerID="26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0"
Jul 23 22:30:14.804103 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.804081    2579 scope.go:117] "RemoveContainer" containerID="26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0"
Jul 23 22:30:14.804370 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:30:14.804348    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0\": container with ID starting with 26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0 not found: ID does not exist" containerID="26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0"
Jul 23 22:30:14.804424 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.804378    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0"} err="failed to get container status \"26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0\": rpc error: code = NotFound desc = could not find container \"26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0\": container with ID starting with 26c9f119309bc231bcf1e7df7763b138002ef6c0410d4c45a7e7ba868d3e88a0 not found: ID does not exist"
Jul 23 22:30:14.815212 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.815179    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"]
Jul 23 22:30:14.821516 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.821486    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8ba6ab/test-model-8ba6ab-kserve-6b654c87cc-qmvkz"]
Jul 23 22:30:14.911611 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:14.911566    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44" podUID="95c98fe1-b72e-40f5-b35c-23f4b669690b" containerName="main" containerID="cri-o://f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346" gracePeriod=45
Jul 23 22:30:15.101772 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.101739    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:15.111568 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.111538    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9a5f046-6b50-42d1-8d7f-7b13166614ae" path="/var/lib/kubelet/pods/e9a5f046-6b50-42d1-8d7f-7b13166614ae/volumes"
Jul 23 22:30:15.162544 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.162518    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:30:15.225085 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.225047    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95c98fe1-b72e-40f5-b35c-23f4b669690b-tls-certs\") pod \"95c98fe1-b72e-40f5-b35c-23f4b669690b\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") "
Jul 23 22:30:15.225085 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.225083    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-home\") pod \"95c98fe1-b72e-40f5-b35c-23f4b669690b\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") "
Jul 23 22:30:15.225312 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.225123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-dshm\") pod \"95c98fe1-b72e-40f5-b35c-23f4b669690b\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") "
Jul 23 22:30:15.225312 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.225170    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86722\" (UniqueName: \"kubernetes.io/projected/95c98fe1-b72e-40f5-b35c-23f4b669690b-kube-api-access-86722\") pod \"95c98fe1-b72e-40f5-b35c-23f4b669690b\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") "
Jul 23 22:30:15.225312 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.225206    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-tmp-dir\") pod \"95c98fe1-b72e-40f5-b35c-23f4b669690b\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") "
Jul 23 22:30:15.225312 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.225236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-model-cache\") pod \"95c98fe1-b72e-40f5-b35c-23f4b669690b\" (UID: \"95c98fe1-b72e-40f5-b35c-23f4b669690b\") "
Jul 23 22:30:15.225531 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.225392    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-home" (OuterVolumeSpecName: "home") pod "95c98fe1-b72e-40f5-b35c-23f4b669690b" (UID: "95c98fe1-b72e-40f5-b35c-23f4b669690b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:15.225590 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.225530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-model-cache" (OuterVolumeSpecName: "model-cache") pod "95c98fe1-b72e-40f5-b35c-23f4b669690b" (UID: "95c98fe1-b72e-40f5-b35c-23f4b669690b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:15.225590 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.225537    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:15.225590 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.225547    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "95c98fe1-b72e-40f5-b35c-23f4b669690b" (UID: "95c98fe1-b72e-40f5-b35c-23f4b669690b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:15.227498 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.227475    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c98fe1-b72e-40f5-b35c-23f4b669690b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95c98fe1-b72e-40f5-b35c-23f4b669690b" (UID: "95c98fe1-b72e-40f5-b35c-23f4b669690b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:30:15.227919 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.227882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-dshm" (OuterVolumeSpecName: "dshm") pod "95c98fe1-b72e-40f5-b35c-23f4b669690b" (UID: "95c98fe1-b72e-40f5-b35c-23f4b669690b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:15.227919 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.227909    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c98fe1-b72e-40f5-b35c-23f4b669690b-kube-api-access-86722" (OuterVolumeSpecName: "kube-api-access-86722") pod "95c98fe1-b72e-40f5-b35c-23f4b669690b" (UID: "95c98fe1-b72e-40f5-b35c-23f4b669690b"). InnerVolumeSpecName "kube-api-access-86722". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:30:15.326356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.326250    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95c98fe1-b72e-40f5-b35c-23f4b669690b-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:15.326356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.326294    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:15.326356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.326309    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86722\" (UniqueName: \"kubernetes.io/projected/95c98fe1-b72e-40f5-b35c-23f4b669690b-kube-api-access-86722\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:15.326356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.326322    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:15.326356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.326336    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95c98fe1-b72e-40f5-b35c-23f4b669690b-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:15.652777 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.652671    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:15.804415 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.804379    2579 generic.go:356] "Generic (PLEG): container finished" podID="95c98fe1-b72e-40f5-b35c-23f4b669690b" containerID="f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346" exitCode=0
Jul 23 22:30:15.804596 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.804424    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"
Jul 23 22:30:15.804596 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.804467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44" event={"ID":"95c98fe1-b72e-40f5-b35c-23f4b669690b","Type":"ContainerDied","Data":"f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346"}
Jul 23 22:30:15.804596 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.804523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44" event={"ID":"95c98fe1-b72e-40f5-b35c-23f4b669690b","Type":"ContainerDied","Data":"9f4bfca2c7b766249c452987a28a3d807c5398939c42ae52e31256d881a8ab90"}
Jul 23 22:30:15.804596 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.804546    2579 scope.go:117] "RemoveContainer" containerID="f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346"
Jul 23 22:30:15.813748 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.813727    2579 scope.go:117] "RemoveContainer" containerID="f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346"
Jul 23 22:30:15.814055 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:30:15.814037    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346\": container with ID starting with f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346 not found: ID does not exist" containerID="f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346"
Jul 23 22:30:15.814124 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.814067    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346"} err="failed to get container status \"f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346\": rpc error: code = NotFound desc = could not find container \"f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346\": container with ID starting with f5e64e3f56afa608fb595ae6bb27b5ba70a7a99930574a22489e0a5acc83b346 not found: ID does not exist"
Jul 23 22:30:15.826077 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.826046    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"]
Jul 23 22:30:15.830005 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:15.829976    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8a3e27/test-model-8a3e27-kserve-6f8578b95c-czp44"]
Jul 23 22:30:17.111815 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:17.111770    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c98fe1-b72e-40f5-b35c-23f4b669690b" path="/var/lib/kubelet/pods/95c98fe1-b72e-40f5-b35c-23f4b669690b/volumes"
Jul 23 22:30:21.920699 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:21.920659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:22.022314 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.022279    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:22.188400 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.188318    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"]
Jul 23 22:30:22.188586 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.188557    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy" containerID="cri-o://9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231" gracePeriod=30
Jul 23 22:30:22.208210 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.208164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:22.459948 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.459907    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:22.587061 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.587019    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"]
Jul 23 22:30:22.587438 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.587419    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12cd4273-173d-4304-8f41-37b9f93ef30e" containerName="maas-api"
Jul 23 22:30:22.587525 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.587440    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="12cd4273-173d-4304-8f41-37b9f93ef30e" containerName="maas-api"
Jul 23 22:30:22.587525 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.587466    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a5f046-6b50-42d1-8d7f-7b13166614ae" containerName="main"
Jul 23 22:30:22.587525 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.587475    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a5f046-6b50-42d1-8d7f-7b13166614ae" containerName="main"
Jul 23 22:30:22.587525 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.587489    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c98fe1-b72e-40f5-b35c-23f4b669690b" containerName="main"
Jul 23 22:30:22.587525 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.587498    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c98fe1-b72e-40f5-b35c-23f4b669690b" containerName="main"
Jul 23 22:30:22.587769 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.587576    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c98fe1-b72e-40f5-b35c-23f4b669690b" containerName="main"
Jul 23 22:30:22.587769 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.587589    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a5f046-6b50-42d1-8d7f-7b13166614ae" containerName="main"
Jul 23 22:30:22.587769 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.587603    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="12cd4273-173d-4304-8f41-37b9f93ef30e" containerName="maas-api"
Jul 23 22:30:22.590668 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.590642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"
Jul 23 22:30:22.592969 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.592937    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lq4mm\""
Jul 23 22:30:22.597815 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.597786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"]
Jul 23 22:30:22.701127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.701088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec361b3d-e57c-475e-887d-1119cb325670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6\" (UID: \"ec361b3d-e57c-475e-887d-1119cb325670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"
Jul 23 22:30:22.802434 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.802345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec361b3d-e57c-475e-887d-1119cb325670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6\" (UID: \"ec361b3d-e57c-475e-887d-1119cb325670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"
Jul 23 22:30:22.803016 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.802995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec361b3d-e57c-475e-887d-1119cb325670-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6\" (UID: \"ec361b3d-e57c-475e-887d-1119cb325670\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"
Jul 23 22:30:22.903198 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:22.903165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"
Jul 23 22:30:23.041550 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:23.041405    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"]
Jul 23 22:30:23.044213 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:30:23.044186    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec361b3d_e57c_475e_887d_1119cb325670.slice/crio-c7168fad004ff8760c70a86688e4bd7be8bad4f20ff8708feb482f509701e16a WatchSource:0}: Error finding container c7168fad004ff8760c70a86688e4bd7be8bad4f20ff8708feb482f509701e16a: Status 404 returned error can't find the container with id c7168fad004ff8760c70a86688e4bd7be8bad4f20ff8708feb482f509701e16a
Jul 23 22:30:23.389072 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:23.389048    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:23.489380 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:23.489346    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:23.842244 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:23.842207    2579 generic.go:356] "Generic (PLEG): container finished" podID="ec361b3d-e57c-475e-887d-1119cb325670" containerID="bade4e0efa0d0848a3c5624e21d805fa99751a7ffae7d4ef2b8d22d2f3629e56" exitCode=0
Jul 23 22:30:23.842411 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:23.842290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6" event={"ID":"ec361b3d-e57c-475e-887d-1119cb325670","Type":"ContainerDied","Data":"bade4e0efa0d0848a3c5624e21d805fa99751a7ffae7d4ef2b8d22d2f3629e56"}
Jul 23 22:30:23.842411 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:23.842345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6" event={"ID":"ec361b3d-e57c-475e-887d-1119cb325670","Type":"ContainerStarted","Data":"c7168fad004ff8760c70a86688e4bd7be8bad4f20ff8708feb482f509701e16a"}
Jul 23 22:30:24.983641 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:24.983612    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"
Jul 23 22:30:25.125272 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:25.125181    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec361b3d-e57c-475e-887d-1119cb325670-maas-api-service-ca\") pod \"ec361b3d-e57c-475e-887d-1119cb325670\" (UID: \"ec361b3d-e57c-475e-887d-1119cb325670\") "
Jul 23 22:30:25.125551 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:25.125527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec361b3d-e57c-475e-887d-1119cb325670-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec361b3d-e57c-475e-887d-1119cb325670" (UID: "ec361b3d-e57c-475e-887d-1119cb325670"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:30:25.226745 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:25.226702    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec361b3d-e57c-475e-887d-1119cb325670-maas-api-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:25.852778 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:25.852744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6" event={"ID":"ec361b3d-e57c-475e-887d-1119cb325670","Type":"ContainerDied","Data":"c7168fad004ff8760c70a86688e4bd7be8bad4f20ff8708feb482f509701e16a"}
Jul 23 22:30:25.852971 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:25.852795    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7168fad004ff8760c70a86688e4bd7be8bad4f20ff8708feb482f509701e16a"
Jul 23 22:30:25.852971 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:25.852758    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"
Jul 23 22:30:27.445275 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.445246    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:30:27.546448 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546349    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-istio-token\") pod \"732355aa-5655-466b-b974-496329d2582c\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") "
Jul 23 22:30:27.546448 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546396    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgbt4\" (UniqueName: \"kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-kube-api-access-tgbt4\") pod \"732355aa-5655-466b-b974-496329d2582c\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") "
Jul 23 22:30:27.546448 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546416    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-certs\") pod \"732355aa-5655-466b-b974-496329d2582c\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") "
Jul 23 22:30:27.546730 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546455    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732355aa-5655-466b-b974-496329d2582c-istio-podinfo\") pod \"732355aa-5655-466b-b974-496329d2582c\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") "
Jul 23 22:30:27.546730 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546502    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-credential-socket\") pod \"732355aa-5655-466b-b974-496329d2582c\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") "
Jul 23 22:30:27.546730 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546537    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-envoy\") pod \"732355aa-5655-466b-b974-496329d2582c\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") "
Jul 23 22:30:27.546730 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546578    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-data\") pod \"732355aa-5655-466b-b974-496329d2582c\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") "
Jul 23 22:30:27.546730 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546601    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-socket\") pod \"732355aa-5655-466b-b974-496329d2582c\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") "
Jul 23 22:30:27.546730 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546633    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732355aa-5655-466b-b974-496329d2582c-istiod-ca-cert\") pod \"732355aa-5655-466b-b974-496329d2582c\" (UID: \"732355aa-5655-466b-b974-496329d2582c\") "
Jul 23 22:30:27.547158 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546780    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "732355aa-5655-466b-b974-496329d2582c" (UID: "732355aa-5655-466b-b974-496329d2582c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:27.547158 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546802    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "732355aa-5655-466b-b974-496329d2582c" (UID: "732355aa-5655-466b-b974-496329d2582c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:27.547158 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546944    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:27.547158 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.546963    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-credential-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:27.547158 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.547029    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "732355aa-5655-466b-b974-496329d2582c" (UID: "732355aa-5655-466b-b974-496329d2582c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:27.547158 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.547136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732355aa-5655-466b-b974-496329d2582c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "732355aa-5655-466b-b974-496329d2582c" (UID: "732355aa-5655-466b-b974-496329d2582c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:30:27.547814 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.547782    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-data" (OuterVolumeSpecName: "istio-data") pod "732355aa-5655-466b-b974-496329d2582c" (UID: "732355aa-5655-466b-b974-496329d2582c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:27.548914 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.548883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-istio-token" (OuterVolumeSpecName: "istio-token") pod "732355aa-5655-466b-b974-496329d2582c" (UID: "732355aa-5655-466b-b974-496329d2582c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:30:27.549025 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.548962    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/732355aa-5655-466b-b974-496329d2582c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "732355aa-5655-466b-b974-496329d2582c" (UID: "732355aa-5655-466b-b974-496329d2582c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 22:30:27.549025 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.548969    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "732355aa-5655-466b-b974-496329d2582c" (UID: "732355aa-5655-466b-b974-496329d2582c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:30:27.549209 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.549191    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-kube-api-access-tgbt4" (OuterVolumeSpecName: "kube-api-access-tgbt4") pod "732355aa-5655-466b-b974-496329d2582c" (UID: "732355aa-5655-466b-b974-496329d2582c"). InnerVolumeSpecName "kube-api-access-tgbt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:30:27.647606 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.647566    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732355aa-5655-466b-b974-496329d2582c-istio-podinfo\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:27.647606 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.647597    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-envoy\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:27.647606 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.647605    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-istio-data\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:27.647606 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.647613    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732355aa-5655-466b-b974-496329d2582c-workload-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:27.647893 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.647621    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732355aa-5655-466b-b974-496329d2582c-istiod-ca-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:27.647893 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.647629    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-istio-token\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:27.647893 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.647637    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgbt4\" (UniqueName: \"kubernetes.io/projected/732355aa-5655-466b-b974-496329d2582c-kube-api-access-tgbt4\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:27.866180 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.866086    2579 generic.go:356] "Generic (PLEG): container finished" podID="732355aa-5655-466b-b974-496329d2582c" containerID="9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231" exitCode=0
Jul 23 22:30:27.866180 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.866141    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"
Jul 23 22:30:27.866180 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.866159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" event={"ID":"732355aa-5655-466b-b974-496329d2582c","Type":"ContainerDied","Data":"9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231"}
Jul 23 22:30:27.866466 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.866227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq" event={"ID":"732355aa-5655-466b-b974-496329d2582c","Type":"ContainerDied","Data":"4e7bcdcb0fca2961143cfa700c0200128e820a9f62b860a5caf70a14cdd94fcb"}
Jul 23 22:30:27.866466 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.866257    2579 scope.go:117] "RemoveContainer" containerID="9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231"
Jul 23 22:30:27.876317 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.876291    2579 scope.go:117] "RemoveContainer" containerID="50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc"
Jul 23 22:30:27.885135 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.885114    2579 scope.go:117] "RemoveContainer" containerID="9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231"
Jul 23 22:30:27.885444 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:30:27.885423    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231\": container with ID starting with 9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231 not found: ID does not exist" containerID="9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231"
Jul 23 22:30:27.885508 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.885454    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231"} err="failed to get container status \"9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231\": rpc error: code = NotFound desc = could not find container \"9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231\": container with ID starting with 9abff5c47f33cd87b13c275cb4d7277eb4ecc9b8ff4a7214de6dc9dcc9969231 not found: ID does not exist"
Jul 23 22:30:27.885508 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.885473    2579 scope.go:117] "RemoveContainer" containerID="50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc"
Jul 23 22:30:27.885726 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:30:27.885707    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc\": container with ID starting with 50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc not found: ID does not exist" containerID="50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc"
Jul 23 22:30:27.885780 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.885735    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc"} err="failed to get container status \"50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc\": rpc error: code = NotFound desc = could not find container \"50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc\": container with ID starting with 50e77c6189d9bf5d15de0c840ca53459eda7e24694990b0c6e30233337b290cc not found: ID does not exist"
Jul 23 22:30:27.888428 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.888404    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"]
Jul 23 22:30:27.890555 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:27.890529    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8ba6ab-openshift-default-6fc6797677-s8nrq"]
Jul 23 22:30:29.112098 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:29.112051    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="732355aa-5655-466b-b974-496329d2582c" path="/var/lib/kubelet/pods/732355aa-5655-466b-b974-496329d2582c/volumes"
Jul 23 22:30:33.165291 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.165256    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"]
Jul 23 22:30:33.165789 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.165478    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc" podUID="d2b14797-c7d7-47a7-96f3-6c0a336dd0e0" containerName="maas-api" containerID="cri-o://69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669" gracePeriod=30
Jul 23 22:30:33.189573 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.189543    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:33.412987 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.412958    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:30:33.598660 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.598616    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-maas-api-tls\") pod \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") "
Jul 23 22:30:33.598849 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.598674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-openshift-service-ca\") pod \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") "
Jul 23 22:30:33.598849 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.598701    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvvpv\" (UniqueName: \"kubernetes.io/projected/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-kube-api-access-fvvpv\") pod \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\" (UID: \"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0\") "
Jul 23 22:30:33.599106 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.599074    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d2b14797-c7d7-47a7-96f3-6c0a336dd0e0" (UID: "d2b14797-c7d7-47a7-96f3-6c0a336dd0e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:30:33.600960 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.600933    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-kube-api-access-fvvpv" (OuterVolumeSpecName: "kube-api-access-fvvpv") pod "d2b14797-c7d7-47a7-96f3-6c0a336dd0e0" (UID: "d2b14797-c7d7-47a7-96f3-6c0a336dd0e0"). InnerVolumeSpecName "kube-api-access-fvvpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:30:33.601080 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.600959    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d2b14797-c7d7-47a7-96f3-6c0a336dd0e0" (UID: "d2b14797-c7d7-47a7-96f3-6c0a336dd0e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:30:33.700230 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.700192    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:33.700230 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.700224    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:33.700440 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.700234    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvvpv\" (UniqueName: \"kubernetes.io/projected/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0-kube-api-access-fvvpv\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:30:33.894989 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.894897    2579 generic.go:356] "Generic (PLEG): container finished" podID="d2b14797-c7d7-47a7-96f3-6c0a336dd0e0" containerID="69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669" exitCode=0
Jul 23 22:30:33.894989 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.894942    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"
Jul 23 22:30:33.895230 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.894988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc" event={"ID":"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0","Type":"ContainerDied","Data":"69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669"}
Jul 23 22:30:33.895230 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.895046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc" event={"ID":"d2b14797-c7d7-47a7-96f3-6c0a336dd0e0","Type":"ContainerDied","Data":"5a1beb66fb0b0c1292feeeed5835e0462700cb5239c3be9612aecb95c114cc21"}
Jul 23 22:30:33.895230 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.895078    2579 scope.go:117] "RemoveContainer" containerID="69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669"
Jul 23 22:30:33.904087 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.904063    2579 scope.go:117] "RemoveContainer" containerID="69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669"
Jul 23 22:30:33.904355 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:30:33.904335    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669\": container with ID starting with 69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669 not found: ID does not exist" containerID="69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669"
Jul 23 22:30:33.904419 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.904363    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669"} err="failed to get container status \"69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669\": rpc error: code = NotFound desc = could not find container \"69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669\": container with ID starting with 69dc81942e453704ac051674768496a67d0a12a54df57205ba71f1cf63995669 not found: ID does not exist"
Jul 23 22:30:33.915999 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.915972    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"]
Jul 23 22:30:33.919436 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:33.919414    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8a3e27-6fdbb9cb98-dkfgc"]
Jul 23 22:30:35.111804 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:35.111771    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b14797-c7d7-47a7-96f3-6c0a336dd0e0" path="/var/lib/kubelet/pods/d2b14797-c7d7-47a7-96f3-6c0a336dd0e0/volumes"
Jul 23 22:30:37.683313 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:37.683274    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"]
Jul 23 22:30:37.685002 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:37.684965    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:37.686204 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:37.686182    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a3e27-k8rb6"]
Jul 23 22:30:38.075788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:38.075755    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:39.112202 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:39.112168    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec361b3d-e57c-475e-887d-1119cb325670" path="/var/lib/kubelet/pods/ec361b3d-e57c-475e-887d-1119cb325670/volumes"
Jul 23 22:30:44.156677 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:44.156624    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:44.257213 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:44.257164    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:44.445680 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:44.445594    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:44.660854 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:44.660817    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:45.028568 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.028529    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:45.318028 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.317937    2579 scope.go:117] "RemoveContainer" containerID="d494ba190007af8ec9739cbfc2a46a42791bedd63befef0e66dd5e79ee58e405"
Jul 23 22:30:45.335646 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.335621    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"]
Jul 23 22:30:45.335976 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.335958    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec361b3d-e57c-475e-887d-1119cb325670" containerName="revoke-keys"
Jul 23 22:30:45.335976 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.335975    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec361b3d-e57c-475e-887d-1119cb325670" containerName="revoke-keys"
Jul 23 22:30:45.336153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.335985    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy"
Jul 23 22:30:45.336153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.335991    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy"
Jul 23 22:30:45.336153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.336000    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b14797-c7d7-47a7-96f3-6c0a336dd0e0" containerName="maas-api"
Jul 23 22:30:45.336153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.336012    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b14797-c7d7-47a7-96f3-6c0a336dd0e0" containerName="maas-api"
Jul 23 22:30:45.336153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.336027    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy"
Jul 23 22:30:45.336153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.336032    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy"
Jul 23 22:30:45.336153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.336096    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy"
Jul 23 22:30:45.336153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.336104    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b14797-c7d7-47a7-96f3-6c0a336dd0e0" containerName="maas-api"
Jul 23 22:30:45.336153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.336111    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="732355aa-5655-466b-b974-496329d2582c" containerName="istio-proxy"
Jul 23 22:30:45.336153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.336115    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec361b3d-e57c-475e-887d-1119cb325670" containerName="revoke-keys"
Jul 23 22:30:45.340307 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.340288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.342686 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.342665    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 22:30:45.342773 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.342702    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c3321b-openshift-default-dockercfg-wsqlj\""
Jul 23 22:30:45.351645 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.351615    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"]
Jul 23 22:30:45.379647 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.379617    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:45.400819 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.400781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-socket\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.400979 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.400823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-credential-socket\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.400979 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.400843    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-podinfo\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.400979 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.400897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wndgl\" (UniqueName: \"kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-kube-api-access-wndgl\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.400979 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.400968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-data\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.401145 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.401017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istiod-ca-cert\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.401145 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.401081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-envoy\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.401145 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.401101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-certs\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.401145 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.401139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-token\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.501566 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.501530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-data\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.501745 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.501580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istiod-ca-cert\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.501745 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.501622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-envoy\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.501745 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.501643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-certs\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.501745 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.501668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-token\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.501745 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.501687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-socket\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.501745 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.501707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-credential-socket\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.501745 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.501731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-podinfo\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.502198 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.501757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wndgl\" (UniqueName: \"kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-kube-api-access-wndgl\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.502198 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.502041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-data\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.502198 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.502127    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-certs\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.502361 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.502278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-socket\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.502361 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.502295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-credential-socket\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.502444 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.502391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istiod-ca-cert\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.504000 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.503981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-envoy\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.504378 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.504361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-podinfo\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.510076 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.510047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-token\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.510186 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.510128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wndgl\" (UniqueName: \"kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-kube-api-access-wndgl\") pod \"e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") " pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.652022 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.651924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:45.794389 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.794352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"]
Jul 23 22:30:45.797792 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:30:45.797762    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399b05d9_f577_4c1b_bf7b_49ee9450ec4a.slice/crio-f1888651f5fe0a580c9540e5e6d7a1ec93dea0112a1176e62815fbfc5ac9eb50 WatchSource:0}: Error finding container f1888651f5fe0a580c9540e5e6d7a1ec93dea0112a1176e62815fbfc5ac9eb50: Status 404 returned error can't find the container with id f1888651f5fe0a580c9540e5e6d7a1ec93dea0112a1176e62815fbfc5ac9eb50
Jul 23 22:30:45.800158 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.800115    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:30:45.800247 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.800208    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:30:45.800291 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.800250    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:30:45.950369 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.950339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" event={"ID":"399b05d9-f577-4c1b-bf7b-49ee9450ec4a","Type":"ContainerStarted","Data":"6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed"}
Jul 23 22:30:45.950565 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.950403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" event={"ID":"399b05d9-f577-4c1b-bf7b-49ee9450ec4a","Type":"ContainerStarted","Data":"f1888651f5fe0a580c9540e5e6d7a1ec93dea0112a1176e62815fbfc5ac9eb50"}
Jul 23 22:30:45.971508 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:45.971443    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" podStartSLOduration=0.971418807 podStartE2EDuration="971.418807ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:30:45.970319415 +0000 UTC m=+2341.360712034" watchObservedRunningTime="2026-07-23 22:30:45.971418807 +0000 UTC m=+2341.361811424"
Jul 23 22:30:46.090516 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:46.090479    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:46.652970 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:46.652924    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:46.657845 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:46.657816    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:46.955104 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:46.955066    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:46.956082 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:46.956054    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:30:47.989698 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:47.989659    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:48.072927 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.072890    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"]
Jul 23 22:30:48.076150 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.076119    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:48.076787 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.076771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:48.078973 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.078950    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c3321b-serving-cert\""
Jul 23 22:30:48.086955 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.086929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"]
Jul 23 22:30:48.125082 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.125025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb119cff-30cd-4b42-b733-9fe5c3b35d76-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:48.125082 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.125081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjk29\" (UniqueName: \"kubernetes.io/projected/bb119cff-30cd-4b42-b733-9fe5c3b35d76-kube-api-access-pjk29\") pod \"maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:48.125344 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.125116    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb119cff-30cd-4b42-b733-9fe5c3b35d76-maas-api-tls\") pod \"maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:48.142383 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.142350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:48.225837 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.225796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb119cff-30cd-4b42-b733-9fe5c3b35d76-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:48.225837 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.225840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjk29\" (UniqueName: \"kubernetes.io/projected/bb119cff-30cd-4b42-b733-9fe5c3b35d76-kube-api-access-pjk29\") pod \"maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:48.226153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.225898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb119cff-30cd-4b42-b733-9fe5c3b35d76-maas-api-tls\") pod \"maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:48.226153 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:30:48.226012    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-serving-cert: secret "maas-api-e2e-api-a-c3321b-serving-cert" not found
Jul 23 22:30:48.226153 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:30:48.226087    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb119cff-30cd-4b42-b733-9fe5c3b35d76-maas-api-tls podName:bb119cff-30cd-4b42-b733-9fe5c3b35d76 nodeName:}" failed. No retries permitted until 2026-07-23 22:30:48.726066727 +0000 UTC m=+2344.116459325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb119cff-30cd-4b42-b733-9fe5c3b35d76-maas-api-tls") pod "maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv" (UID: "bb119cff-30cd-4b42-b733-9fe5c3b35d76") : secret "maas-api-e2e-api-a-c3321b-serving-cert" not found
Jul 23 22:30:48.226586 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.226559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb119cff-30cd-4b42-b733-9fe5c3b35d76-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:48.240852 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.240778    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjk29\" (UniqueName: \"kubernetes.io/projected/bb119cff-30cd-4b42-b733-9fe5c3b35d76-kube-api-access-pjk29\") pod \"maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:48.621148 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.621054    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:48.730607 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.730568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb119cff-30cd-4b42-b733-9fe5c3b35d76-maas-api-tls\") pod \"maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:48.733112 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.733085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb119cff-30cd-4b42-b733-9fe5c3b35d76-maas-api-tls\") pod \"maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:48.778193 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.778146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:48.878638 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.878558    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:48.989195 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:48.989153    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:49.128457 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:49.128429    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"]
Jul 23 22:30:49.130897 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:30:49.130851    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb119cff_30cd_4b42_b733_9fe5c3b35d76.slice/crio-a12cdf58def01bb74f507f60d16acfabbc1979dec70ff7ac815fca89fbc0ef52 WatchSource:0}: Error finding container a12cdf58def01bb74f507f60d16acfabbc1979dec70ff7ac815fca89fbc0ef52: Status 404 returned error can't find the container with id a12cdf58def01bb74f507f60d16acfabbc1979dec70ff7ac815fca89fbc0ef52
Jul 23 22:30:49.969025 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:49.968990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv" event={"ID":"bb119cff-30cd-4b42-b733-9fe5c3b35d76","Type":"ContainerStarted","Data":"07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840"}
Jul 23 22:30:49.969178 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:49.969043    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv" event={"ID":"bb119cff-30cd-4b42-b733-9fe5c3b35d76","Type":"ContainerStarted","Data":"a12cdf58def01bb74f507f60d16acfabbc1979dec70ff7ac815fca89fbc0ef52"}
Jul 23 22:30:49.969178 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:49.969096    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:49.987406 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:49.987351    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv" podStartSLOduration=1.5189045719999998 podStartE2EDuration="1.987333734s" totalImagesPullingTime="468.429162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:30:49.985591062 +0000 UTC m=+2345.375983678" watchObservedRunningTime="2026-07-23 22:30:49.987333734 +0000 UTC m=+2345.377726349"
Jul 23 22:30:52.534183 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:52.534151    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:53.026879 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:53.026840    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:53.344415 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:53.344329    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:53.418429 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:53.418394    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:55.979683 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:55.979648    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:30:56.532117 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.532082    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:56.615018 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.614983    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"]
Jul 23 22:30:56.618843 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.618817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:30:56.621153 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.621126    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-fc9d5e-serving-cert\""
Jul 23 22:30:56.625721 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.625690    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:56.632422 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.632395    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"]
Jul 23 22:30:56.698688 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.698656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:56.700842 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.700815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-maas-api-tls\") pod \"maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:30:56.700988 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.700902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-openshift-service-ca\") pod \"maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:30:56.700988 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.700935    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvlz\" (UniqueName: \"kubernetes.io/projected/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-kube-api-access-dgvlz\") pod \"maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:30:56.799545 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.799454    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:56.801635 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.801609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-openshift-service-ca\") pod \"maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:30:56.801777 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.801659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvlz\" (UniqueName: \"kubernetes.io/projected/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-kube-api-access-dgvlz\") pod \"maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:30:56.801777 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.801722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-maas-api-tls\") pod \"maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:30:56.802401 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.802381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-openshift-service-ca\") pod \"maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:30:56.804407 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.804382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-maas-api-tls\") pod \"maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:30:56.811369 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.811341    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvlz\" (UniqueName: \"kubernetes.io/projected/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-kube-api-access-dgvlz\") pod \"maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:30:56.930227 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:56.930185    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:30:57.112581 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:57.112554    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"]
Jul 23 22:30:57.114873 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:30:57.114830    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf9d7f1_e517_45f0_8f4f_e65651887cbe.slice/crio-a79e706e7e2a8d23cf34613fa6b7dde969aca72108235adec3da21495f4ef462 WatchSource:0}: Error finding container a79e706e7e2a8d23cf34613fa6b7dde969aca72108235adec3da21495f4ef462: Status 404 returned error can't find the container with id a79e706e7e2a8d23cf34613fa6b7dde969aca72108235adec3da21495f4ef462
Jul 23 22:30:57.382077 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:57.381944    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:57.482322 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:57.482252    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:30:58.007442 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:58.007407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7" event={"ID":"4bf9d7f1-e517-45f0-8f4f-e65651887cbe","Type":"ContainerStarted","Data":"25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039"}
Jul 23 22:30:58.007631 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:58.007462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7" event={"ID":"4bf9d7f1-e517-45f0-8f4f-e65651887cbe","Type":"ContainerStarted","Data":"a79e706e7e2a8d23cf34613fa6b7dde969aca72108235adec3da21495f4ef462"}
Jul 23 22:30:58.007631 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:30:58.007540    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:31:01.025463 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:01.025425    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:02.998041 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:02.997989    2579 patch_prober.go:28] interesting pod/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 23 22:31:02.998530 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:02.998083    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 23 22:31:04.016838 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:04.016811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:31:04.033686 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:04.033632    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7" podStartSLOduration=7.708363765 podStartE2EDuration="8.033617126s" totalImagesPullingTime="325.253361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:30:58.02745834 +0000 UTC m=+2353.417850957" watchObservedRunningTime="2026-07-23 22:31:04.033617126 +0000 UTC m=+2359.424009743"
Jul 23 22:31:04.036620 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:04.036594    2579 generic.go:356] "Generic (PLEG): container finished" podID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerID="6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed" exitCode=0
Jul 23 22:31:04.036748 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:04.036669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" event={"ID":"399b05d9-f577-4c1b-bf7b-49ee9450ec4a","Type":"ContainerDied","Data":"6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed"}
Jul 23 22:31:04.037115 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:04.037099    2579 scope.go:117] "RemoveContainer" containerID="6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed"
Jul 23 22:31:04.038107 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:04.038076    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:31:04.038171 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:04.038145    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:31:04.038216 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:04.038175    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:31:05.045428 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:05.045397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" event={"ID":"399b05d9-f577-4c1b-bf7b-49ee9450ec4a","Type":"ContainerStarted","Data":"2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2"}
Jul 23 22:31:05.652572 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:05.652535    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:31:05.652843 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:05.652820    2579 patch_prober.go:28] interesting pod/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 23 22:31:05.652935 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:05.652909    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 23 22:31:06.653214 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:06.653168    2579 patch_prober.go:28] interesting pod/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 23 22:31:06.653654 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:06.653246    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 23 22:31:07.652829 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:07.652792    2579 patch_prober.go:28] interesting pod/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 23 22:31:07.653046 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:07.652894    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 23 22:31:08.653325 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:08.653284    2579 patch_prober.go:28] interesting pod/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 23 22:31:08.653808 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:08.653349    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 23 22:31:09.656445 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:09.656403    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:31:09.656903 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:09.656831    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:31:09.657671 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:09.657649    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:31:09.967004 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:09.966975    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:10.070045 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:10.070006    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:20.613122 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:20.613084    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:20.757670 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:20.757643    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:24.879567 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:24.879520    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"]
Jul 23 22:31:24.880081 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:24.879810    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv" podUID="bb119cff-30cd-4b42-b733-9fe5c3b35d76" containerName="maas-api" containerID="cri-o://07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840" gracePeriod=30
Jul 23 22:31:24.902795 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:24.902766    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:25.129996 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.129936    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:31:25.135643 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.135615    2579 generic.go:356] "Generic (PLEG): container finished" podID="bb119cff-30cd-4b42-b733-9fe5c3b35d76" containerID="07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840" exitCode=0
Jul 23 22:31:25.135765 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.135655    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"
Jul 23 22:31:25.135765 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.135689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv" event={"ID":"bb119cff-30cd-4b42-b733-9fe5c3b35d76","Type":"ContainerDied","Data":"07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840"}
Jul 23 22:31:25.135765 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.135736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv" event={"ID":"bb119cff-30cd-4b42-b733-9fe5c3b35d76","Type":"ContainerDied","Data":"a12cdf58def01bb74f507f60d16acfabbc1979dec70ff7ac815fca89fbc0ef52"}
Jul 23 22:31:25.135765 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.135758    2579 scope.go:117] "RemoveContainer" containerID="07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840"
Jul 23 22:31:25.143936 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.143917    2579 scope.go:117] "RemoveContainer" containerID="07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840"
Jul 23 22:31:25.144201 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:31:25.144182    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840\": container with ID starting with 07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840 not found: ID does not exist" containerID="07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840"
Jul 23 22:31:25.144253 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.144212    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840"} err="failed to get container status \"07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840\": rpc error: code = NotFound desc = could not find container \"07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840\": container with ID starting with 07ee91e4d974b3b57fa0c95e912370dfe9d3a2f21906e944afd81a5e3f91f840 not found: ID does not exist"
Jul 23 22:31:25.263546 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.263507    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb119cff-30cd-4b42-b733-9fe5c3b35d76-openshift-service-ca\") pod \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") "
Jul 23 22:31:25.263721 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.263561    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjk29\" (UniqueName: \"kubernetes.io/projected/bb119cff-30cd-4b42-b733-9fe5c3b35d76-kube-api-access-pjk29\") pod \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") "
Jul 23 22:31:25.263721 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.263597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb119cff-30cd-4b42-b733-9fe5c3b35d76-maas-api-tls\") pod \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\" (UID: \"bb119cff-30cd-4b42-b733-9fe5c3b35d76\") "
Jul 23 22:31:25.263956 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.263931    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb119cff-30cd-4b42-b733-9fe5c3b35d76-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb119cff-30cd-4b42-b733-9fe5c3b35d76" (UID: "bb119cff-30cd-4b42-b733-9fe5c3b35d76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:31:25.265828 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.265805    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb119cff-30cd-4b42-b733-9fe5c3b35d76-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb119cff-30cd-4b42-b733-9fe5c3b35d76" (UID: "bb119cff-30cd-4b42-b733-9fe5c3b35d76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:31:25.265944 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.265923    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb119cff-30cd-4b42-b733-9fe5c3b35d76-kube-api-access-pjk29" (OuterVolumeSpecName: "kube-api-access-pjk29") pod "bb119cff-30cd-4b42-b733-9fe5c3b35d76" (UID: "bb119cff-30cd-4b42-b733-9fe5c3b35d76"). InnerVolumeSpecName "kube-api-access-pjk29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:31:25.365084 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.365045    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb119cff-30cd-4b42-b733-9fe5c3b35d76-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:25.365084 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.365077    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb119cff-30cd-4b42-b733-9fe5c3b35d76-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:25.365084 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.365088    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjk29\" (UniqueName: \"kubernetes.io/projected/bb119cff-30cd-4b42-b733-9fe5c3b35d76-kube-api-access-pjk29\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:25.456778 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.456742    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"]
Jul 23 22:31:25.462747 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:25.462716    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3321b-66b5c79ddd-4mgjv"]
Jul 23 22:31:27.112005 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:27.111967    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb119cff-30cd-4b42-b733-9fe5c3b35d76" path="/var/lib/kubelet/pods/bb119cff-30cd-4b42-b733-9fe5c3b35d76/volumes"
Jul 23 22:31:29.311789 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:29.311760    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:29.713260 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:29.713231    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:35.872727 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:35.872692    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:35.972105 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:35.972067    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:36.195020 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.194932    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"]
Jul 23 22:31:36.195184 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.195162    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy" containerID="cri-o://2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2" gracePeriod=30
Jul 23 22:31:36.246852 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.246816    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:36.407525 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.407489    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:36.522599 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.522563    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"]
Jul 23 22:31:36.522910 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.522898    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb119cff-30cd-4b42-b733-9fe5c3b35d76" containerName="maas-api"
Jul 23 22:31:36.522959 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.522912    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb119cff-30cd-4b42-b733-9fe5c3b35d76" containerName="maas-api"
Jul 23 22:31:36.522995 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.522979    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb119cff-30cd-4b42-b733-9fe5c3b35d76" containerName="maas-api"
Jul 23 22:31:36.526869 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.526837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"
Jul 23 22:31:36.529225 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.529203    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lq4mm\""
Jul 23 22:31:36.534637 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.534607    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"]
Jul 23 22:31:36.549631 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.549593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/383b2f81-0e11-459d-8107-22b005d9a976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg\" (UID: \"383b2f81-0e11-459d-8107-22b005d9a976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"
Jul 23 22:31:36.650974 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.650928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/383b2f81-0e11-459d-8107-22b005d9a976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg\" (UID: \"383b2f81-0e11-459d-8107-22b005d9a976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"
Jul 23 22:31:36.651577 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.651551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/383b2f81-0e11-459d-8107-22b005d9a976-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg\" (UID: \"383b2f81-0e11-459d-8107-22b005d9a976\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"
Jul 23 22:31:36.839505 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.839413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"
Jul 23 22:31:36.977739 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:36.977679    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"]
Jul 23 22:31:36.979963 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:31:36.979933    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383b2f81_0e11_459d_8107_22b005d9a976.slice/crio-a88cafefba34539d48bb11107b4dd683019f79f9740208f8e0e03bd5584c5608 WatchSource:0}: Error finding container a88cafefba34539d48bb11107b4dd683019f79f9740208f8e0e03bd5584c5608: Status 404 returned error can't find the container with id a88cafefba34539d48bb11107b4dd683019f79f9740208f8e0e03bd5584c5608
Jul 23 22:31:37.197457 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:37.197421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg" event={"ID":"383b2f81-0e11-459d-8107-22b005d9a976","Type":"ContainerStarted","Data":"a88cafefba34539d48bb11107b4dd683019f79f9740208f8e0e03bd5584c5608"}
Jul 23 22:31:38.203110 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:38.203072    2579 generic.go:356] "Generic (PLEG): container finished" podID="383b2f81-0e11-459d-8107-22b005d9a976" containerID="fd7520265aa6d37bf04b207d4c6002e326a56437614820fe534d14db5f949a10" exitCode=0
Jul 23 22:31:38.203474 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:38.203134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg" event={"ID":"383b2f81-0e11-459d-8107-22b005d9a976","Type":"ContainerDied","Data":"fd7520265aa6d37bf04b207d4c6002e326a56437614820fe534d14db5f949a10"}
Jul 23 22:31:38.578695 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:38.578607    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:38.679098 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:38.679066    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:39.338895 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:39.338842    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"
Jul 23 22:31:39.374008 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:39.373974    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/383b2f81-0e11-459d-8107-22b005d9a976-maas-api-service-ca\") pod \"383b2f81-0e11-459d-8107-22b005d9a976\" (UID: \"383b2f81-0e11-459d-8107-22b005d9a976\") "
Jul 23 22:31:39.374339 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:39.374313    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383b2f81-0e11-459d-8107-22b005d9a976-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "383b2f81-0e11-459d-8107-22b005d9a976" (UID: "383b2f81-0e11-459d-8107-22b005d9a976"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:31:39.475932 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:39.475810    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/383b2f81-0e11-459d-8107-22b005d9a976-maas-api-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:39.533495 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:39.533462    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:39.634415 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:39.634377    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:39.659381 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:39.659346    2579 patch_prober.go:28] interesting pod/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 23 22:31:39.659530 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:39.659411    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 23 22:31:40.212746 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:40.212715    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"
Jul 23 22:31:40.212974 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:40.212705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg" event={"ID":"383b2f81-0e11-459d-8107-22b005d9a976","Type":"ContainerDied","Data":"a88cafefba34539d48bb11107b4dd683019f79f9740208f8e0e03bd5584c5608"}
Jul 23 22:31:40.212974 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:40.212856    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a88cafefba34539d48bb11107b4dd683019f79f9740208f8e0e03bd5584c5608"
Jul 23 22:31:40.292620 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:40.292585    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:40.378366 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:40.378327    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:41.252982 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.252947    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:41.354326 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.354289    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:41.451596 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.451570    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:31:41.495126 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495029    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-socket\") pod \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") "
Jul 23 22:31:41.495126 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495082    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-podinfo\") pod \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") "
Jul 23 22:31:41.495126 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495114    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istiod-ca-cert\") pod \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") "
Jul 23 22:31:41.495410 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-credential-socket\") pod \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") "
Jul 23 22:31:41.495410 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495152    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-data\") pod \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") "
Jul 23 22:31:41.495410 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495179    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-envoy\") pod \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") "
Jul 23 22:31:41.495410 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495209    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-certs\") pod \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") "
Jul 23 22:31:41.495410 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495235    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-token\") pod \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") "
Jul 23 22:31:41.495410 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495310    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wndgl\" (UniqueName: \"kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-kube-api-access-wndgl\") pod \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\" (UID: \"399b05d9-f577-4c1b-bf7b-49ee9450ec4a\") "
Jul 23 22:31:41.495410 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "399b05d9-f577-4c1b-bf7b-49ee9450ec4a" (UID: "399b05d9-f577-4c1b-bf7b-49ee9450ec4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:31:41.495792 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495482    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "399b05d9-f577-4c1b-bf7b-49ee9450ec4a" (UID: "399b05d9-f577-4c1b-bf7b-49ee9450ec4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:31:41.495792 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "399b05d9-f577-4c1b-bf7b-49ee9450ec4a" (UID: "399b05d9-f577-4c1b-bf7b-49ee9450ec4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:31:41.495792 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495607    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:41.495792 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495624    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-credential-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:41.495792 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.495699    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "399b05d9-f577-4c1b-bf7b-49ee9450ec4a" (UID: "399b05d9-f577-4c1b-bf7b-49ee9450ec4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:31:41.496582 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.496548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-data" (OuterVolumeSpecName: "istio-data") pod "399b05d9-f577-4c1b-bf7b-49ee9450ec4a" (UID: "399b05d9-f577-4c1b-bf7b-49ee9450ec4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:31:41.497654 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.497623    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "399b05d9-f577-4c1b-bf7b-49ee9450ec4a" (UID: "399b05d9-f577-4c1b-bf7b-49ee9450ec4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 22:31:41.497654 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.497636    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-token" (OuterVolumeSpecName: "istio-token") pod "399b05d9-f577-4c1b-bf7b-49ee9450ec4a" (UID: "399b05d9-f577-4c1b-bf7b-49ee9450ec4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:31:41.497798 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.497682    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "399b05d9-f577-4c1b-bf7b-49ee9450ec4a" (UID: "399b05d9-f577-4c1b-bf7b-49ee9450ec4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:31:41.497798 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.497689    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-kube-api-access-wndgl" (OuterVolumeSpecName: "kube-api-access-wndgl") pod "399b05d9-f577-4c1b-bf7b-49ee9450ec4a" (UID: "399b05d9-f577-4c1b-bf7b-49ee9450ec4a"). InnerVolumeSpecName "kube-api-access-wndgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:31:41.596532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.596493    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-podinfo\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:41.596532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.596525    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istiod-ca-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:41.596532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.596534    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-data\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:41.596532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.596542    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-envoy\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:41.596799 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.596550    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-workload-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:41.596799 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.596558    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-istio-token\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:41.596799 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:41.596566    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wndgl\" (UniqueName: \"kubernetes.io/projected/399b05d9-f577-4c1b-bf7b-49ee9450ec4a-kube-api-access-wndgl\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:42.047819 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.047781    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:42.145021 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.144982    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:42.224366 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.224336    2579 generic.go:356] "Generic (PLEG): container finished" podID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerID="2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2" exitCode=0
Jul 23 22:31:42.224556 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.224387    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"
Jul 23 22:31:42.224556 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.224397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" event={"ID":"399b05d9-f577-4c1b-bf7b-49ee9450ec4a","Type":"ContainerDied","Data":"2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2"}
Jul 23 22:31:42.224556 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.224446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw" event={"ID":"399b05d9-f577-4c1b-bf7b-49ee9450ec4a","Type":"ContainerDied","Data":"f1888651f5fe0a580c9540e5e6d7a1ec93dea0112a1176e62815fbfc5ac9eb50"}
Jul 23 22:31:42.224556 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.224471    2579 scope.go:117] "RemoveContainer" containerID="2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2"
Jul 23 22:31:42.234628 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.234607    2579 scope.go:117] "RemoveContainer" containerID="6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed"
Jul 23 22:31:42.243082 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.243061    2579 scope.go:117] "RemoveContainer" containerID="2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2"
Jul 23 22:31:42.243362 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:31:42.243344    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2\": container with ID starting with 2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2 not found: ID does not exist" containerID="2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2"
Jul 23 22:31:42.243413 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.243372    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2"} err="failed to get container status \"2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2\": rpc error: code = NotFound desc = could not find container \"2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2\": container with ID starting with 2d36bdd907eca3192dbb2dcfcac9a15acf90c669165add63a559559064748ec2 not found: ID does not exist"
Jul 23 22:31:42.243413 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.243390    2579 scope.go:117] "RemoveContainer" containerID="6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed"
Jul 23 22:31:42.243667 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:31:42.243648    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed\": container with ID starting with 6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed not found: ID does not exist" containerID="6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed"
Jul 23 22:31:42.243729 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.243676    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed"} err="failed to get container status \"6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed\": rpc error: code = NotFound desc = could not find container \"6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed\": container with ID starting with 6aeb93dd90c7500c7b2640580a1222a00a9bdce0d9df0db4f7f5c425361c38ed not found: ID does not exist"
Jul 23 22:31:42.246853 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.246829    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"]
Jul 23 22:31:42.252002 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.251977    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c3321b-openshift-default-db7c4d5c9-jzrrw"]
Jul 23 22:31:42.997923 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:42.997893    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:43.098008 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:43.097974    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:43.111595 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:43.111558    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" path="/var/lib/kubelet/pods/399b05d9-f577-4c1b-bf7b-49ee9450ec4a/volumes"
Jul 23 22:31:43.761244 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:43.761207    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:43.862472 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:43.862419    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:44.667816 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:44.667785    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:44.768579 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:44.768533    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:45.162509 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:45.162478    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:31:45.164835 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:45.164808    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:31:45.455364 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:45.455337    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:45.555820 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:45.555791    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:46.465361 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:46.465331    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:46.564842 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:46.564811    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:47.256013 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:47.255983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:47.354889 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:47.354843    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:52.505045 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.505006    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"]
Jul 23 22:31:52.505547 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.505247    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7" podUID="4bf9d7f1-e517-45f0-8f4f-e65651887cbe" containerName="maas-api" containerID="cri-o://25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039" gracePeriod=30
Jul 23 22:31:52.530080 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.530050    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:52.746051 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.746026    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:31:52.796914 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.796786    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgvlz\" (UniqueName: \"kubernetes.io/projected/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-kube-api-access-dgvlz\") pod \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") "
Jul 23 22:31:52.796914 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.796907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-maas-api-tls\") pod \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") "
Jul 23 22:31:52.797150 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.796970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-openshift-service-ca\") pod \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\" (UID: \"4bf9d7f1-e517-45f0-8f4f-e65651887cbe\") "
Jul 23 22:31:52.797356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.797330    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4bf9d7f1-e517-45f0-8f4f-e65651887cbe" (UID: "4bf9d7f1-e517-45f0-8f4f-e65651887cbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:31:52.799171 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.799148    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4bf9d7f1-e517-45f0-8f4f-e65651887cbe" (UID: "4bf9d7f1-e517-45f0-8f4f-e65651887cbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:31:52.799258 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.799226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-kube-api-access-dgvlz" (OuterVolumeSpecName: "kube-api-access-dgvlz") pod "4bf9d7f1-e517-45f0-8f4f-e65651887cbe" (UID: "4bf9d7f1-e517-45f0-8f4f-e65651887cbe"). InnerVolumeSpecName "kube-api-access-dgvlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:31:52.897571 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.897530    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgvlz\" (UniqueName: \"kubernetes.io/projected/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-kube-api-access-dgvlz\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:52.897571 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.897569    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:52.897571 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:52.897579    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bf9d7f1-e517-45f0-8f4f-e65651887cbe-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:31:53.274221 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:53.274184    2579 generic.go:356] "Generic (PLEG): container finished" podID="4bf9d7f1-e517-45f0-8f4f-e65651887cbe" containerID="25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039" exitCode=0
Jul 23 22:31:53.274416 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:53.274230    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"
Jul 23 22:31:53.274416 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:53.274268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7" event={"ID":"4bf9d7f1-e517-45f0-8f4f-e65651887cbe","Type":"ContainerDied","Data":"25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039"}
Jul 23 22:31:53.274416 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:53.274321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7" event={"ID":"4bf9d7f1-e517-45f0-8f4f-e65651887cbe","Type":"ContainerDied","Data":"a79e706e7e2a8d23cf34613fa6b7dde969aca72108235adec3da21495f4ef462"}
Jul 23 22:31:53.274416 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:53.274343    2579 scope.go:117] "RemoveContainer" containerID="25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039"
Jul 23 22:31:53.284820 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:53.284802    2579 scope.go:117] "RemoveContainer" containerID="25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039"
Jul 23 22:31:53.285121 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:31:53.285099    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039\": container with ID starting with 25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039 not found: ID does not exist" containerID="25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039"
Jul 23 22:31:53.285198 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:53.285135    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039"} err="failed to get container status \"25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039\": rpc error: code = NotFound desc = could not find container \"25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039\": container with ID starting with 25b1b996ae4ac302698df47bc581fda3ff8dd4b96a52a0da38b9c3182ced2039 not found: ID does not exist"
Jul 23 22:31:53.291042 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:53.291016    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"]
Jul 23 22:31:53.294759 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:53.294736    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fc9d5e-5959c8fc55-t2qb7"]
Jul 23 22:31:55.111744 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:55.111712    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bf9d7f1-e517-45f0-8f4f-e65651887cbe" path="/var/lib/kubelet/pods/4bf9d7f1-e517-45f0-8f4f-e65651887cbe/volumes"
Jul 23 22:31:56.605969 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:56.605936    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:56.609875 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:56.609839    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"]
Jul 23 22:31:56.612173 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:56.612149    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fc9d5e-zpvlg"]
Jul 23 22:31:57.008413 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:57.008382    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:31:57.111302 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:31:57.111266    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="383b2f81-0e11-459d-8107-22b005d9a976" path="/var/lib/kubelet/pods/383b2f81-0e11-459d-8107-22b005d9a976/volumes"
Jul 23 22:32:02.987798 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:02.987759    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:03.049097 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:03.049061    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:03.269683 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:03.269600    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:03.488045 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:03.488012    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:03.839219 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:03.839187    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:04.137814 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.137732    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"]
Jul 23 22:32:04.138188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.138050    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy"
Jul 23 22:32:04.138188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.138062    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy"
Jul 23 22:32:04.138188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.138073    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bf9d7f1-e517-45f0-8f4f-e65651887cbe" containerName="maas-api"
Jul 23 22:32:04.138188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.138079    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bf9d7f1-e517-45f0-8f4f-e65651887cbe" containerName="maas-api"
Jul 23 22:32:04.138188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.138090    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="383b2f81-0e11-459d-8107-22b005d9a976" containerName="revoke-keys"
Jul 23 22:32:04.138188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.138096    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="383b2f81-0e11-459d-8107-22b005d9a976" containerName="revoke-keys"
Jul 23 22:32:04.138188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.138159    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="383b2f81-0e11-459d-8107-22b005d9a976" containerName="revoke-keys"
Jul 23 22:32:04.138188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.138166    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bf9d7f1-e517-45f0-8f4f-e65651887cbe" containerName="maas-api"
Jul 23 22:32:04.138188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.138176    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy"
Jul 23 22:32:04.138188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.138182    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy"
Jul 23 22:32:04.142500 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.142478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.145388 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.145365    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-a1850d-openshift-default-dockercfg-qv94j\""
Jul 23 22:32:04.145659 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.145643    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 22:32:04.156047 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.156015    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"]
Jul 23 22:32:04.166931 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.166903    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:04.301186 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.301153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-certs\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.301359 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.301206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a473ba-7800-4e53-baca-43db45064258-istiod-ca-cert\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.301359 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.301272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnpt6\" (UniqueName: \"kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-kube-api-access-xnpt6\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.301359 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.301308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-socket\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.301359 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.301326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-envoy\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.301359 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.301350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-istio-token\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.301538 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.301399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a473ba-7800-4e53-baca-43db45064258-istio-podinfo\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.301538 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.301427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-credential-socket\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.301538 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.301472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-data\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.401994 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.401884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-certs\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.401994 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.401959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a473ba-7800-4e53-baca-43db45064258-istiod-ca-cert\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.401994 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.401997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnpt6\" (UniqueName: \"kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-kube-api-access-xnpt6\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.402284 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.402025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-socket\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.402284 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.402049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-envoy\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.402284 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.402080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-istio-token\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.402284 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.402118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a473ba-7800-4e53-baca-43db45064258-istio-podinfo\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.402284 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.402158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-credential-socket\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.402284 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.402194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-data\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.402610 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.402286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-certs\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.402610 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.402352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-socket\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.402610 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.402585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-credential-socket\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.402712 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.402614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-data\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.402748 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.402722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a473ba-7800-4e53-baca-43db45064258-istiod-ca-cert\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.404530 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.404508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-envoy\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.404655 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.404636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a473ba-7800-4e53-baca-43db45064258-istio-podinfo\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.411577 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.411548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnpt6\" (UniqueName: \"kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-kube-api-access-xnpt6\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.412696 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.412672    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-istio-token\") pod \"e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") " pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.452993 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.452960    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:04.561709 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.561677    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:04.592011 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.591823    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"]
Jul 23 22:32:04.594655 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:32:04.594616    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a473ba_7800_4e53_baca_43db45064258.slice/crio-d16ec4d8f207c938e0d7dbd4ce0391990c949d18250cb083cfb33fa78af1139b WatchSource:0}: Error finding container d16ec4d8f207c938e0d7dbd4ce0391990c949d18250cb083cfb33fa78af1139b: Status 404 returned error can't find the container with id d16ec4d8f207c938e0d7dbd4ce0391990c949d18250cb083cfb33fa78af1139b
Jul 23 22:32:04.596619 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.596586    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:32:04.596700 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.596674    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:32:04.596743 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:04.596707    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:32:05.327961 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:05.327849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" event={"ID":"00a473ba-7800-4e53-baca-43db45064258","Type":"ContainerStarted","Data":"997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184"}
Jul 23 22:32:05.327961 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:05.327929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" event={"ID":"00a473ba-7800-4e53-baca-43db45064258","Type":"ContainerStarted","Data":"d16ec4d8f207c938e0d7dbd4ce0391990c949d18250cb083cfb33fa78af1139b"}
Jul 23 22:32:05.350380 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:05.350326    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" podStartSLOduration=1.350308748 podStartE2EDuration="1.350308748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:32:05.349274711 +0000 UTC m=+2420.739667330" watchObservedRunningTime="2026-07-23 22:32:05.350308748 +0000 UTC m=+2420.740701364"
Jul 23 22:32:05.453234 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:05.453200    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:05.458277 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:05.458248    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:06.332937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:06.332906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:06.333940 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:06.333921    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:07.342494 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.342450    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:07.427889 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.427819    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"]
Jul 23 22:32:07.428259 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.428240    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy"
Jul 23 22:32:07.428354 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.428261    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="399b05d9-f577-4c1b-bf7b-49ee9450ec4a" containerName="istio-proxy"
Jul 23 22:32:07.431109 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.431087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:07.432572 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.432526    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:07.433308 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.433290    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a1850d-serving-cert\""
Jul 23 22:32:07.449601 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.449562    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"]
Jul 23 22:32:07.514508 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.514469    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:07.531347 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.531308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a7bbce-775b-47ea-9b08-fa9e7018af7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:07.531539 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.531353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a7bbce-775b-47ea-9b08-fa9e7018af7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:07.531539 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.531438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2895z\" (UniqueName: \"kubernetes.io/projected/20a7bbce-775b-47ea-9b08-fa9e7018af7b-kube-api-access-2895z\") pod \"maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:07.632094 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.631996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a7bbce-775b-47ea-9b08-fa9e7018af7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:07.632094 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.632036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a7bbce-775b-47ea-9b08-fa9e7018af7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:07.632094 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.632088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2895z\" (UniqueName: \"kubernetes.io/projected/20a7bbce-775b-47ea-9b08-fa9e7018af7b-kube-api-access-2895z\") pod \"maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:07.632314 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:32:07.632237    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-serving-cert: secret "maas-api-e2e-shared-a-a1850d-serving-cert" not found
Jul 23 22:32:07.632347 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:32:07.632327    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a7bbce-775b-47ea-9b08-fa9e7018af7b-maas-api-tls podName:20a7bbce-775b-47ea-9b08-fa9e7018af7b nodeName:}" failed. No retries permitted until 2026-07-23 22:32:08.132302518 +0000 UTC m=+2423.522695134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20a7bbce-775b-47ea-9b08-fa9e7018af7b-maas-api-tls") pod "maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9" (UID: "20a7bbce-775b-47ea-9b08-fa9e7018af7b") : secret "maas-api-e2e-shared-a-a1850d-serving-cert" not found
Jul 23 22:32:07.632732 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.632712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a7bbce-775b-47ea-9b08-fa9e7018af7b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:07.647962 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.647923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2895z\" (UniqueName: \"kubernetes.io/projected/20a7bbce-775b-47ea-9b08-fa9e7018af7b-kube-api-access-2895z\") pod \"maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:07.746650 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:07.746612    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:08.041427 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:08.041389    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:08.136394 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:08.136355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a7bbce-775b-47ea-9b08-fa9e7018af7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:08.139144 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:08.139114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a7bbce-775b-47ea-9b08-fa9e7018af7b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:08.144837 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:08.144809    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:08.348332 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:08.348243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:08.484484 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:08.484446    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"]
Jul 23 22:32:08.488423 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:32:08.488381    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a7bbce_775b_47ea_9b08_fa9e7018af7b.slice/crio-aa494138d1217e2e0dc3e8e365c316bae0e5798d73c4d4ff97c0d7a74b4004ae WatchSource:0}: Error finding container aa494138d1217e2e0dc3e8e365c316bae0e5798d73c4d4ff97c0d7a74b4004ae: Status 404 returned error can't find the container with id aa494138d1217e2e0dc3e8e365c316bae0e5798d73c4d4ff97c0d7a74b4004ae
Jul 23 22:32:09.347168 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:09.347120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9" event={"ID":"20a7bbce-775b-47ea-9b08-fa9e7018af7b","Type":"ContainerStarted","Data":"0758c8d945b07d346c9ef6b4f47cb088318f2fcab09aaff8e485a454972bdbf2"}
Jul 23 22:32:09.347339 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:09.347183    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:09.347339 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:09.347207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9" event={"ID":"20a7bbce-775b-47ea-9b08-fa9e7018af7b","Type":"ContainerStarted","Data":"aa494138d1217e2e0dc3e8e365c316bae0e5798d73c4d4ff97c0d7a74b4004ae"}
Jul 23 22:32:09.366013 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:09.365961    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9" podStartSLOduration=1.97901333 podStartE2EDuration="2.365943478s" totalImagesPullingTime="386.930148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:32:09.365141844 +0000 UTC m=+2424.755534460" watchObservedRunningTime="2026-07-23 22:32:09.365943478 +0000 UTC m=+2424.756336094"
Jul 23 22:32:11.860749 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:11.860698    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:12.211262 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:12.211228    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:12.534407 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:12.534300    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:12.793590 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:12.793505    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:15.127760 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:15.127721    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:15.357736 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:15.357702    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:32:15.939759 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:15.939728    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:16.021538 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.021494    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"]
Jul 23 22:32:16.025375 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.025355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:16.027568 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.027541    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-091ba5-serving-cert\""
Jul 23 22:32:16.028104 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.028080    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:16.034397 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.034371    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"]
Jul 23 22:32:16.099771 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.099738    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:16.215007 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.214973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xpsc\" (UniqueName: \"kubernetes.io/projected/3f414145-66e2-4979-a331-99ebf804f9a1-kube-api-access-2xpsc\") pod \"maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:16.215435 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.215036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f414145-66e2-4979-a331-99ebf804f9a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:16.215435 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.215082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f414145-66e2-4979-a331-99ebf804f9a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:16.254479 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.254447    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:16.315520 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.315483    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f414145-66e2-4979-a331-99ebf804f9a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:16.315725 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.315550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xpsc\" (UniqueName: \"kubernetes.io/projected/3f414145-66e2-4979-a331-99ebf804f9a1-kube-api-access-2xpsc\") pod \"maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:16.315725 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.315587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f414145-66e2-4979-a331-99ebf804f9a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:16.316233 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.316207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f414145-66e2-4979-a331-99ebf804f9a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:16.318157 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.318131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f414145-66e2-4979-a331-99ebf804f9a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:16.323275 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.323253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xpsc\" (UniqueName: \"kubernetes.io/projected/3f414145-66e2-4979-a331-99ebf804f9a1-kube-api-access-2xpsc\") pod \"maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:16.336979 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.336955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:16.518608 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.518579    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"]
Jul 23 22:32:16.521051 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:32:16.521022    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f414145_66e2_4979_a331_99ebf804f9a1.slice/crio-1c9aaa7de4f5ea278d485143d1e5ab8ca9b93337d230e1f499e44ffbc8b4fe04 WatchSource:0}: Error finding container 1c9aaa7de4f5ea278d485143d1e5ab8ca9b93337d230e1f499e44ffbc8b4fe04: Status 404 returned error can't find the container with id 1c9aaa7de4f5ea278d485143d1e5ab8ca9b93337d230e1f499e44ffbc8b4fe04
Jul 23 22:32:16.607185 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.607147    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:16.708037 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:16.708002    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:17.401380 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:17.401341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs" event={"ID":"3f414145-66e2-4979-a331-99ebf804f9a1","Type":"ContainerStarted","Data":"1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698"}
Jul 23 22:32:17.401852 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:17.401400    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:17.401852 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:17.401424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs" event={"ID":"3f414145-66e2-4979-a331-99ebf804f9a1","Type":"ContainerStarted","Data":"1c9aaa7de4f5ea278d485143d1e5ab8ca9b93337d230e1f499e44ffbc8b4fe04"}
Jul 23 22:32:17.419263 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:17.419214    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs" podStartSLOduration=1.042449246 podStartE2EDuration="1.419197714s" totalImagesPullingTime="376.748468ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:32:17.416650873 +0000 UTC m=+2432.807043492" watchObservedRunningTime="2026-07-23 22:32:17.419197714 +0000 UTC m=+2432.809590329"
Jul 23 22:32:20.436147 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:20.436101    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:21.334614 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:21.334574    2579 patch_prober.go:28] interesting pod/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 23 22:32:21.334795 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:21.334641    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 23 22:32:21.420571 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:21.420538    2579 generic.go:356] "Generic (PLEG): container finished" podID="00a473ba-7800-4e53-baca-43db45064258" containerID="997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184" exitCode=0
Jul 23 22:32:21.420748 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:21.420618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" event={"ID":"00a473ba-7800-4e53-baca-43db45064258","Type":"ContainerDied","Data":"997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184"}
Jul 23 22:32:21.421110 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:21.421092    2579 scope.go:117] "RemoveContainer" containerID="997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184"
Jul 23 22:32:21.422140 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:21.422107    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:32:21.422260 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:21.422198    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:32:21.422260 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:21.422245    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 23 22:32:22.426681 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:22.426647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" event={"ID":"00a473ba-7800-4e53-baca-43db45064258","Type":"ContainerStarted","Data":"fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232"}
Jul 23 22:32:22.453594 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:22.453558    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:22.453805 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:22.453751    2579 patch_prober.go:28] interesting pod/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 23 22:32:22.453885 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:22.453799    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 23 22:32:23.411237 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:23.411206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:32:23.453619 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:23.453583    2579 patch_prober.go:28] interesting pod/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 23 22:32:23.454036 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:23.453653    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 23 22:32:24.453603 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:24.453570    2579 patch_prober.go:28] interesting pod/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 23 22:32:24.453806 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:24.453632    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 23 22:32:25.453320 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:25.453285    2579 patch_prober.go:28] interesting pod/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 23 22:32:25.453509 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:25.453342    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 23 22:32:26.457340 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:26.457308    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:26.457742 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:26.457584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:26.458219 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:26.458203    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:32:29.920302 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:29.920245    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:30.018467 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:30.018416    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:37.617878 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.617827    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"]
Jul 23 22:32:37.620201 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.620183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.622347 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.622326    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a1850d\"/\"openshift-service-ca.crt\""
Jul 23 22:32:37.623188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.623163    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a1850d\"/\"default-dockercfg-zdrb5\""
Jul 23 22:32:37.623188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.623181    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a1850d\"/\"kube-root-ca.crt\""
Jul 23 22:32:37.623354 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.623190    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a1850d\"/\"auth-test-model-a1850d-kserve-self-signed-certs\""
Jul 23 22:32:37.634545 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.634515    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"]
Jul 23 22:32:37.649251 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.649220    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:37.708023 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.707985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-dshm\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.708200 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.708086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcxs\" (UniqueName: \"kubernetes.io/projected/d9c46788-2501-4418-ac8f-1b3bedd3542d-kube-api-access-rrcxs\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.708200 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.708109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-model-cache\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.708200 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.708129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-tmp-dir\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.708200 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.708181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-home\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.708351 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.708204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9c46788-2501-4418-ac8f-1b3bedd3542d-tls-certs\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.808954 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.808919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-tmp-dir\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.809127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.808975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-home\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.809127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.808995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9c46788-2501-4418-ac8f-1b3bedd3542d-tls-certs\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.809127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.809030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-dshm\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.809127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.809097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcxs\" (UniqueName: \"kubernetes.io/projected/d9c46788-2501-4418-ac8f-1b3bedd3542d-kube-api-access-rrcxs\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.809127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.809124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-model-cache\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.809413 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.809388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-tmp-dir\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.809413 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.809404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-home\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.809504 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.809479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-model-cache\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.811442 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.811417    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-dshm\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.811747 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.811726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9c46788-2501-4418-ac8f-1b3bedd3542d-tls-certs\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.818167 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.818141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcxs\" (UniqueName: \"kubernetes.io/projected/d9c46788-2501-4418-ac8f-1b3bedd3542d-kube-api-access-rrcxs\") pod \"auth-test-model-a1850d-kserve-6bff67fc4c-vf62k\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") " pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:37.930060 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:37.929964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:38.067961 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:38.067929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"]
Jul 23 22:32:38.071091 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:32:38.071063    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9c46788_2501_4418_ac8f_1b3bedd3542d.slice/crio-57669787b03b92127b7e558319dbfe1f055acea656a5b608580d8a449b86f741 WatchSource:0}: Error finding container 57669787b03b92127b7e558319dbfe1f055acea656a5b608580d8a449b86f741: Status 404 returned error can't find the container with id 57669787b03b92127b7e558319dbfe1f055acea656a5b608580d8a449b86f741
Jul 23 22:32:38.498566 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:38.498528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k" event={"ID":"d9c46788-2501-4418-ac8f-1b3bedd3542d","Type":"ContainerStarted","Data":"498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403"}
Jul 23 22:32:38.499016 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:38.498575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k" event={"ID":"d9c46788-2501-4418-ac8f-1b3bedd3542d","Type":"ContainerStarted","Data":"57669787b03b92127b7e558319dbfe1f055acea656a5b608580d8a449b86f741"}
Jul 23 22:32:38.517413 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:38.517345    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k" podStartSLOduration=1.5173303059999999 podStartE2EDuration="1.517330306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:32:38.516480669 +0000 UTC m=+2453.906873286" watchObservedRunningTime="2026-07-23 22:32:38.517330306 +0000 UTC m=+2453.907722921"
Jul 23 22:32:39.051755 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:39.051722    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:39.151808 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:39.151771    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:45.371540 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:45.371503    2579 scope.go:117] "RemoveContainer" containerID="ee5d61cebb7944897971fd5db9d7a20936c5f18491a8a08262a54e504df7dc7f"
Jul 23 22:32:47.930290 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:47.930252    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:47.930728 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:47.930314    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:47.943018 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:47.942991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:48.552754 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:48.552722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:32:58.426313 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.426275    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"]
Jul 23 22:32:58.428741 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.428719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.432250 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.432220    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"kube-root-ca.crt\""
Jul 23 22:32:58.432409 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.432273    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"auth-test-model-091ba5-kserve-self-signed-certs\""
Jul 23 22:32:58.432409 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.432328    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"default-dockercfg-gjrm9\""
Jul 23 22:32:58.432550 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.432435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"]
Jul 23 22:32:58.432550 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.432513    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"openshift-service-ca.crt\""
Jul 23 22:32:58.441323 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.441298    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:32:58.597740 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.597698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tmp-dir\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.597937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.597831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-model-cache\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.597937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.597889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgv49\" (UniqueName: \"kubernetes.io/projected/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-kube-api-access-kgv49\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.597937 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.597917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tls-certs\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.598055 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.597962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-home\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.598055 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.597997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-dshm\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.699574 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.699473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgv49\" (UniqueName: \"kubernetes.io/projected/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-kube-api-access-kgv49\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.699574 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.699520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tls-certs\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.699785 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.699580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-home\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.699785 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.699625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-dshm\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.699785 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.699663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tmp-dir\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.699785 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.699778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-model-cache\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.700182 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.700160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-home\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.700259 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.700180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tmp-dir\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.700298 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.700253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-model-cache\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.702085 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.702068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-dshm\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.702262 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.702192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tls-certs\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.708625 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.708585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgv49\" (UniqueName: \"kubernetes.io/projected/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-kube-api-access-kgv49\") pod \"auth-test-model-091ba5-kserve-b4dcfc899-ns947\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") " pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.739496 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.739459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:32:58.880430 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:58.880395    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"]
Jul 23 22:32:58.883582 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:32:58.883557    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc329fd9_89ac_4f63_9b63_fa9c4db70f0b.slice/crio-016c11b1321a370bcff86a2de93c332022b02fb6bfee6048cc23112134f44024 WatchSource:0}: Error finding container 016c11b1321a370bcff86a2de93c332022b02fb6bfee6048cc23112134f44024: Status 404 returned error can't find the container with id 016c11b1321a370bcff86a2de93c332022b02fb6bfee6048cc23112134f44024
Jul 23 22:32:59.588845 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:59.588810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947" event={"ID":"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b","Type":"ContainerStarted","Data":"22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e"}
Jul 23 22:32:59.589271 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:59.588879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947" event={"ID":"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b","Type":"ContainerStarted","Data":"016c11b1321a370bcff86a2de93c332022b02fb6bfee6048cc23112134f44024"}
Jul 23 22:32:59.605734 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:32:59.605679    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947" podStartSLOduration=1.605664332 podStartE2EDuration="1.605664332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:32:59.603391555 +0000 UTC m=+2474.993784170" watchObservedRunningTime="2026-07-23 22:32:59.605664332 +0000 UTC m=+2474.996056948"
Jul 23 22:33:08.740555 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:33:08.740493    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:33:08.740555 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:33:08.740570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:33:08.753813 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:33:08.753779    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:33:09.647349 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:33:09.647317    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:33:27.910952 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:33:27.910914    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:34:17.067682 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:34:17.067652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:34:41.526663 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:34:41.526621    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:35:14.000059 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:13.999978    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:35:14.098966 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:14.098928    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:35:17.714775 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:17.714739    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"]
Jul 23 22:35:17.716623 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:17.716583    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:35:18.648623 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:18.648587    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"]
Jul 23 22:35:18.670441 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:18.670398    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:35:27.017303 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.017267    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"]
Jul 23 22:35:27.020344 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.020323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.022763 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.022743    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a1850d\"/\"sub-test-model-a1850d-kserve-self-signed-certs\""
Jul 23 22:35:27.030322 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.030297    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"]
Jul 23 22:35:27.044663 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.044629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c446f1a-ccde-4b7a-97f6-73794a12233d-tls-certs\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.044800 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.044668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-home\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.044800 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.044693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkz6r\" (UniqueName: \"kubernetes.io/projected/5c446f1a-ccde-4b7a-97f6-73794a12233d-kube-api-access-wkz6r\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.044800 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.044767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-dshm\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.044941 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.044828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-tmp-dir\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.044980 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.044944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-model-cache\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.106604 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.106562    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:35:27.146050 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.146004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-model-cache\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.146790 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.146423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c446f1a-ccde-4b7a-97f6-73794a12233d-tls-certs\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.146984 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.146824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-home\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.147065 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.146993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-model-cache\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.147065 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.147020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkz6r\" (UniqueName: \"kubernetes.io/projected/5c446f1a-ccde-4b7a-97f6-73794a12233d-kube-api-access-wkz6r\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.147168 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.147100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-dshm\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.147223 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.147202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-tmp-dir\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.147388 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.147367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-home\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.148057 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.148033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-tmp-dir\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.150820 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.150589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c446f1a-ccde-4b7a-97f6-73794a12233d-tls-certs\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.156948 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.156914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-dshm\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.157107 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.157086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkz6r\" (UniqueName: \"kubernetes.io/projected/5c446f1a-ccde-4b7a-97f6-73794a12233d-kube-api-access-wkz6r\") pod \"sub-test-model-a1850d-kserve-788459945c-rmcjf\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") " pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.329947 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.329821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:27.469708 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:27.469679    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"]
Jul 23 22:35:27.472286 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:35:27.472255    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c446f1a_ccde_4b7a_97f6_73794a12233d.slice/crio-c9b8a1048f943b6ac8d9c557fe03965c38619975cbd638ac46b0aaff15f7515d WatchSource:0}: Error finding container c9b8a1048f943b6ac8d9c557fe03965c38619975cbd638ac46b0aaff15f7515d: Status 404 returned error can't find the container with id c9b8a1048f943b6ac8d9c557fe03965c38619975cbd638ac46b0aaff15f7515d
Jul 23 22:35:28.251504 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:28.251470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf" event={"ID":"5c446f1a-ccde-4b7a-97f6-73794a12233d","Type":"ContainerStarted","Data":"7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1"}
Jul 23 22:35:28.251917 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:28.251512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf" event={"ID":"5c446f1a-ccde-4b7a-97f6-73794a12233d","Type":"ContainerStarted","Data":"c9b8a1048f943b6ac8d9c557fe03965c38619975cbd638ac46b0aaff15f7515d"}
Jul 23 22:35:28.269108 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:28.269049    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf" podStartSLOduration=2.269032207 podStartE2EDuration="2.269032207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:35:28.268594442 +0000 UTC m=+2623.658987055" watchObservedRunningTime="2026-07-23 22:35:28.269032207 +0000 UTC m=+2623.659424840"
Jul 23 22:35:28.830312 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:28.830275    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:35:32.734960 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:32.734913    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k" podUID="d9c46788-2501-4418-ac8f-1b3bedd3542d" containerName="main" containerID="cri-o://498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403" gracePeriod=45
Jul 23 22:35:32.985719 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:32.985646    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:35:33.105180 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.105132    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-home\") pod \"d9c46788-2501-4418-ac8f-1b3bedd3542d\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") "
Jul 23 22:35:33.105386 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.105210    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrcxs\" (UniqueName: \"kubernetes.io/projected/d9c46788-2501-4418-ac8f-1b3bedd3542d-kube-api-access-rrcxs\") pod \"d9c46788-2501-4418-ac8f-1b3bedd3542d\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") "
Jul 23 22:35:33.105386 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.105252    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9c46788-2501-4418-ac8f-1b3bedd3542d-tls-certs\") pod \"d9c46788-2501-4418-ac8f-1b3bedd3542d\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") "
Jul 23 22:35:33.105386 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.105286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-model-cache\") pod \"d9c46788-2501-4418-ac8f-1b3bedd3542d\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") "
Jul 23 22:35:33.105386 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.105319    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-dshm\") pod \"d9c46788-2501-4418-ac8f-1b3bedd3542d\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") "
Jul 23 22:35:33.105386 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.105373    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-tmp-dir\") pod \"d9c46788-2501-4418-ac8f-1b3bedd3542d\" (UID: \"d9c46788-2501-4418-ac8f-1b3bedd3542d\") "
Jul 23 22:35:33.105660 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.105457    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-home" (OuterVolumeSpecName: "home") pod "d9c46788-2501-4418-ac8f-1b3bedd3542d" (UID: "d9c46788-2501-4418-ac8f-1b3bedd3542d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:35:33.105660 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.105642    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:33.105660 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.105609    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9c46788-2501-4418-ac8f-1b3bedd3542d" (UID: "d9c46788-2501-4418-ac8f-1b3bedd3542d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:35:33.105822 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.105674    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9c46788-2501-4418-ac8f-1b3bedd3542d" (UID: "d9c46788-2501-4418-ac8f-1b3bedd3542d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:35:33.107712 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.107681    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c46788-2501-4418-ac8f-1b3bedd3542d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9c46788-2501-4418-ac8f-1b3bedd3542d" (UID: "d9c46788-2501-4418-ac8f-1b3bedd3542d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:35:33.107827 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.107723    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c46788-2501-4418-ac8f-1b3bedd3542d-kube-api-access-rrcxs" (OuterVolumeSpecName: "kube-api-access-rrcxs") pod "d9c46788-2501-4418-ac8f-1b3bedd3542d" (UID: "d9c46788-2501-4418-ac8f-1b3bedd3542d"). InnerVolumeSpecName "kube-api-access-rrcxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:35:33.107827 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.107796    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-dshm" (OuterVolumeSpecName: "dshm") pod "d9c46788-2501-4418-ac8f-1b3bedd3542d" (UID: "d9c46788-2501-4418-ac8f-1b3bedd3542d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:35:33.206507 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.206462    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:33.206507 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.206496    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrcxs\" (UniqueName: \"kubernetes.io/projected/d9c46788-2501-4418-ac8f-1b3bedd3542d-kube-api-access-rrcxs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:33.206507 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.206512    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9c46788-2501-4418-ac8f-1b3bedd3542d-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:33.206756 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.206524    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:33.206756 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.206535    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9c46788-2501-4418-ac8f-1b3bedd3542d-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:33.275913 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.275808    2579 generic.go:356] "Generic (PLEG): container finished" podID="d9c46788-2501-4418-ac8f-1b3bedd3542d" containerID="498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403" exitCode=0
Jul 23 22:35:33.275913 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.275885    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"
Jul 23 22:35:33.276117 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.275903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k" event={"ID":"d9c46788-2501-4418-ac8f-1b3bedd3542d","Type":"ContainerDied","Data":"498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403"}
Jul 23 22:35:33.276117 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.275958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k" event={"ID":"d9c46788-2501-4418-ac8f-1b3bedd3542d","Type":"ContainerDied","Data":"57669787b03b92127b7e558319dbfe1f055acea656a5b608580d8a449b86f741"}
Jul 23 22:35:33.276117 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.275984    2579 scope.go:117] "RemoveContainer" containerID="498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403"
Jul 23 22:35:33.284698 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.284679    2579 scope.go:117] "RemoveContainer" containerID="498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403"
Jul 23 22:35:33.285023 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:35:33.285001    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403\": container with ID starting with 498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403 not found: ID does not exist" containerID="498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403"
Jul 23 22:35:33.285119 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.285031    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403"} err="failed to get container status \"498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403\": rpc error: code = NotFound desc = could not find container \"498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403\": container with ID starting with 498a7903734d182be9121f10c5f9c7c24100fa251215c998c58dcedbf6d7c403 not found: ID does not exist"
Jul 23 22:35:33.291667 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.291643    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"]
Jul 23 22:35:33.295284 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.295262    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/auth-test-model-a1850d-kserve-6bff67fc4c-vf62k"]
Jul 23 22:35:33.669832 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.669743    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947" podUID="cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" containerName="main" containerID="cri-o://22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e" gracePeriod=45
Jul 23 22:35:33.922521 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:33.922458    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:35:34.013925 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.013885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-home\") pod \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") "
Jul 23 22:35:34.014132 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.013943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-model-cache\") pod \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") "
Jul 23 22:35:34.014132 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.014009    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tls-certs\") pod \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") "
Jul 23 22:35:34.014132 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.014030    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-dshm\") pod \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") "
Jul 23 22:35:34.014132 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.014060    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tmp-dir\") pod \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") "
Jul 23 22:35:34.014346 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.014139    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgv49\" (UniqueName: \"kubernetes.io/projected/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-kube-api-access-kgv49\") pod \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\" (UID: \"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b\") "
Jul 23 22:35:34.014346 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.014234    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-home" (OuterVolumeSpecName: "home") pod "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" (UID: "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:35:34.014346 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.014249    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" (UID: "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:35:34.014346 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.014332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" (UID: "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:35:34.014498 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.014450    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:34.014498 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.014473    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:34.014498 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.014490    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:34.016324 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.016289    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" (UID: "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:35:34.016446 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.016363    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-dshm" (OuterVolumeSpecName: "dshm") pod "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" (UID: "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:35:34.016446 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.016378    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-kube-api-access-kgv49" (OuterVolumeSpecName: "kube-api-access-kgv49") pod "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" (UID: "cc329fd9-89ac-4f63-9b63-fa9c4db70f0b"). InnerVolumeSpecName "kube-api-access-kgv49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:35:34.115270 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.115229    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:34.115270 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.115264    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:34.115270 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.115274    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgv49\" (UniqueName: \"kubernetes.io/projected/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b-kube-api-access-kgv49\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:35:34.282411 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.282374    2579 generic.go:356] "Generic (PLEG): container finished" podID="cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" containerID="22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e" exitCode=0
Jul 23 22:35:34.282606 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.282416    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"
Jul 23 22:35:34.282606 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.282447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947" event={"ID":"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b","Type":"ContainerDied","Data":"22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e"}
Jul 23 22:35:34.282606 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.282489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947" event={"ID":"cc329fd9-89ac-4f63-9b63-fa9c4db70f0b","Type":"ContainerDied","Data":"016c11b1321a370bcff86a2de93c332022b02fb6bfee6048cc23112134f44024"}
Jul 23 22:35:34.282606 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.282511    2579 scope.go:117] "RemoveContainer" containerID="22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e"
Jul 23 22:35:34.292260 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.292236    2579 scope.go:117] "RemoveContainer" containerID="22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e"
Jul 23 22:35:34.292550 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:35:34.292527    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e\": container with ID starting with 22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e not found: ID does not exist" containerID="22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e"
Jul 23 22:35:34.292622 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.292561    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e"} err="failed to get container status \"22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e\": rpc error: code = NotFound desc = could not find container \"22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e\": container with ID starting with 22fddebeaf4f9d0ee19a8deb9df29ae76dcc3b4dd37c79634195837378d8a98e not found: ID does not exist"
Jul 23 22:35:34.304934 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.304904    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"]
Jul 23 22:35:34.308555 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:34.308530    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/auth-test-model-091ba5-kserve-b4dcfc899-ns947"]
Jul 23 22:35:35.112237 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:35.112205    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" path="/var/lib/kubelet/pods/cc329fd9-89ac-4f63-9b63-fa9c4db70f0b/volumes"
Jul 23 22:35:35.112611 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:35.112584    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c46788-2501-4418-ac8f-1b3bedd3542d" path="/var/lib/kubelet/pods/d9c46788-2501-4418-ac8f-1b3bedd3542d/volumes"
Jul 23 22:35:37.330400 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:37.330362    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:37.330806 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:37.330415    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:37.342795 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:37.342771    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:35:38.313401 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:35:38.313366    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:36:07.423241 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.423199    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"]
Jul 23 22:36:07.423788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.423647    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" containerName="main"
Jul 23 22:36:07.423788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.423663    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" containerName="main"
Jul 23 22:36:07.423788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.423680    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c46788-2501-4418-ac8f-1b3bedd3542d" containerName="main"
Jul 23 22:36:07.423788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.423686    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c46788-2501-4418-ac8f-1b3bedd3542d" containerName="main"
Jul 23 22:36:07.423788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.423733    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc329fd9-89ac-4f63-9b63-fa9c4db70f0b" containerName="main"
Jul 23 22:36:07.423788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.423747    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c46788-2501-4418-ac8f-1b3bedd3542d" containerName="main"
Jul 23 22:36:07.426707 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.426685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.429181 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.429158    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"kube-root-ca.crt\""
Jul 23 22:36:07.429900 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.429835    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"sub-test-model-091ba5-kserve-self-signed-certs\""
Jul 23 22:36:07.430039 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.429933    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"openshift-service-ca.crt\""
Jul 23 22:36:07.430039 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.429967    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"default-dockercfg-gjrm9\""
Jul 23 22:36:07.436416 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.436388    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"]
Jul 23 22:36:07.457394 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.457351    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:36:07.532289 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.532250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/574e2f38-f339-4519-a89f-dbfa28d408c1-tls-certs\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.532289 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.532288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkvh\" (UniqueName: \"kubernetes.io/projected/574e2f38-f339-4519-a89f-dbfa28d408c1-kube-api-access-6vkvh\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.532531 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.532351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-tmp-dir\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.532531 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.532405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-home\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.532531 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.532433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-dshm\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.532531 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.532454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-model-cache\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.633508 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.633469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-home\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.633508 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.633504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-dshm\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.633769 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.633521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-model-cache\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.633769 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.633566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/574e2f38-f339-4519-a89f-dbfa28d408c1-tls-certs\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.633769 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.633582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkvh\" (UniqueName: \"kubernetes.io/projected/574e2f38-f339-4519-a89f-dbfa28d408c1-kube-api-access-6vkvh\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.633769 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.633617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-tmp-dir\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.634019 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.633953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-home\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.634019 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.633962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-tmp-dir\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.634133 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.634022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-model-cache\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.635915 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.635894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-dshm\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.636187 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.636169    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/574e2f38-f339-4519-a89f-dbfa28d408c1-tls-certs\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.642264 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.642241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkvh\" (UniqueName: \"kubernetes.io/projected/574e2f38-f339-4519-a89f-dbfa28d408c1-kube-api-access-6vkvh\") pod \"sub-test-model-091ba5-kserve-fbffcb747-77gxl\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") " pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.737622 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.737563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:07.883513 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:07.883481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"]
Jul 23 22:36:07.885008 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:36:07.884983    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod574e2f38_f339_4519_a89f_dbfa28d408c1.slice/crio-9e917316604d03f762aa10122c978563dc701d417a3fe632039702e51a2f0806 WatchSource:0}: Error finding container 9e917316604d03f762aa10122c978563dc701d417a3fe632039702e51a2f0806: Status 404 returned error can't find the container with id 9e917316604d03f762aa10122c978563dc701d417a3fe632039702e51a2f0806
Jul 23 22:36:08.434074 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:08.434042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl" event={"ID":"574e2f38-f339-4519-a89f-dbfa28d408c1","Type":"ContainerStarted","Data":"4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044"}
Jul 23 22:36:08.434074 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:08.434087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl" event={"ID":"574e2f38-f339-4519-a89f-dbfa28d408c1","Type":"ContainerStarted","Data":"9e917316604d03f762aa10122c978563dc701d417a3fe632039702e51a2f0806"}
Jul 23 22:36:08.452015 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:08.451959    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl" podStartSLOduration=1.4519419089999999 podStartE2EDuration="1.451941909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:36:08.450568253 +0000 UTC m=+2663.840960872" watchObservedRunningTime="2026-07-23 22:36:08.451941909 +0000 UTC m=+2663.842334524"
Jul 23 22:36:17.738548 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:17.738508    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:17.739080 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:17.738570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:17.751191 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:17.751167    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:18.490612 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:18.490580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:36:39.505064 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:39.504980    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:36:39.605871 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:39.605836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:36:45.188874 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:45.188825    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:36:45.193061 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:45.193032    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rs2g2_295b3136-a93c-4ff3-8e39-cb60ac4a1fa0/ovn-acl-logging/0.log"
Jul 23 22:36:45.423922 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:45.423886    2579 scope.go:117] "RemoveContainer" containerID="bade4e0efa0d0848a3c5624e21d805fa99751a7ffae7d4ef2b8d22d2f3629e56"
Jul 23 22:36:58.946535 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:58.946496    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"]
Jul 23 22:36:58.969402 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:58.969371    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:36:59.901801 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:59.901762    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"]
Jul 23 22:36:59.921741 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:36:59.921707    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:37:06.657020 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:06.656983    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:37:13.967013 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:13.966950    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf" podUID="5c446f1a-ccde-4b7a-97f6-73794a12233d" containerName="main" containerID="cri-o://7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1" gracePeriod=45
Jul 23 22:37:14.223670 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.223603    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:37:14.352276 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.352244    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-dshm\") pod \"5c446f1a-ccde-4b7a-97f6-73794a12233d\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") "
Jul 23 22:37:14.352455 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.352289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkz6r\" (UniqueName: \"kubernetes.io/projected/5c446f1a-ccde-4b7a-97f6-73794a12233d-kube-api-access-wkz6r\") pod \"5c446f1a-ccde-4b7a-97f6-73794a12233d\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") "
Jul 23 22:37:14.352455 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.352306    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-tmp-dir\") pod \"5c446f1a-ccde-4b7a-97f6-73794a12233d\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") "
Jul 23 22:37:14.352455 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.352330    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-home\") pod \"5c446f1a-ccde-4b7a-97f6-73794a12233d\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") "
Jul 23 22:37:14.352455 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.352379    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-model-cache\") pod \"5c446f1a-ccde-4b7a-97f6-73794a12233d\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") "
Jul 23 22:37:14.352455 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.352423    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c446f1a-ccde-4b7a-97f6-73794a12233d-tls-certs\") pod \"5c446f1a-ccde-4b7a-97f6-73794a12233d\" (UID: \"5c446f1a-ccde-4b7a-97f6-73794a12233d\") "
Jul 23 22:37:14.352737 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.352592    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c446f1a-ccde-4b7a-97f6-73794a12233d" (UID: "5c446f1a-ccde-4b7a-97f6-73794a12233d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:37:14.352737 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.352666    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-home" (OuterVolumeSpecName: "home") pod "5c446f1a-ccde-4b7a-97f6-73794a12233d" (UID: "5c446f1a-ccde-4b7a-97f6-73794a12233d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:37:14.352828 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.352741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c446f1a-ccde-4b7a-97f6-73794a12233d" (UID: "5c446f1a-ccde-4b7a-97f6-73794a12233d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:37:14.352828 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.352755    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:14.354699 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.354671    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-dshm" (OuterVolumeSpecName: "dshm") pod "5c446f1a-ccde-4b7a-97f6-73794a12233d" (UID: "5c446f1a-ccde-4b7a-97f6-73794a12233d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:37:14.354699 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.354669    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c446f1a-ccde-4b7a-97f6-73794a12233d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c446f1a-ccde-4b7a-97f6-73794a12233d" (UID: "5c446f1a-ccde-4b7a-97f6-73794a12233d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:37:14.354911 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.354735    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c446f1a-ccde-4b7a-97f6-73794a12233d-kube-api-access-wkz6r" (OuterVolumeSpecName: "kube-api-access-wkz6r") pod "5c446f1a-ccde-4b7a-97f6-73794a12233d" (UID: "5c446f1a-ccde-4b7a-97f6-73794a12233d"). InnerVolumeSpecName "kube-api-access-wkz6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:37:14.453491 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.453450    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:14.453491 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.453483    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkz6r\" (UniqueName: \"kubernetes.io/projected/5c446f1a-ccde-4b7a-97f6-73794a12233d-kube-api-access-wkz6r\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:14.453491 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.453493    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:14.453491 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.453501    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c446f1a-ccde-4b7a-97f6-73794a12233d-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:14.453759 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.453511    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c446f1a-ccde-4b7a-97f6-73794a12233d-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:14.738478 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.738437    2579 generic.go:356] "Generic (PLEG): container finished" podID="5c446f1a-ccde-4b7a-97f6-73794a12233d" containerID="7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1" exitCode=0
Jul 23 22:37:14.738664 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.738499    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"
Jul 23 22:37:14.738664 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.738521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf" event={"ID":"5c446f1a-ccde-4b7a-97f6-73794a12233d","Type":"ContainerDied","Data":"7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1"}
Jul 23 22:37:14.738664 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.738571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf" event={"ID":"5c446f1a-ccde-4b7a-97f6-73794a12233d","Type":"ContainerDied","Data":"c9b8a1048f943b6ac8d9c557fe03965c38619975cbd638ac46b0aaff15f7515d"}
Jul 23 22:37:14.738664 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.738588    2579 scope.go:117] "RemoveContainer" containerID="7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1"
Jul 23 22:37:14.747872 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.747839    2579 scope.go:117] "RemoveContainer" containerID="7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1"
Jul 23 22:37:14.748143 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:37:14.748121    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1\": container with ID starting with 7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1 not found: ID does not exist" containerID="7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1"
Jul 23 22:37:14.748203 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.748152    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1"} err="failed to get container status \"7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1\": rpc error: code = NotFound desc = could not find container \"7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1\": container with ID starting with 7ef574b806eeb2bb63d14047f43eaa56b7da1e6289e4d4ede8143e4b45a4dde1 not found: ID does not exist"
Jul 23 22:37:14.759006 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.758979    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"]
Jul 23 22:37:14.762716 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.762683    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/sub-test-model-a1850d-kserve-788459945c-rmcjf"]
Jul 23 22:37:14.922021 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:14.921953    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl" podUID="574e2f38-f339-4519-a89f-dbfa28d408c1" containerName="main" containerID="cri-o://4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044" gracePeriod=45
Jul 23 22:37:15.113111 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.113074    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c446f1a-ccde-4b7a-97f6-73794a12233d" path="/var/lib/kubelet/pods/5c446f1a-ccde-4b7a-97f6-73794a12233d/volumes"
Jul 23 22:37:15.171337 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.171307    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:37:15.261508 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.261473    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vkvh\" (UniqueName: \"kubernetes.io/projected/574e2f38-f339-4519-a89f-dbfa28d408c1-kube-api-access-6vkvh\") pod \"574e2f38-f339-4519-a89f-dbfa28d408c1\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") "
Jul 23 22:37:15.261723 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.261517    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-model-cache\") pod \"574e2f38-f339-4519-a89f-dbfa28d408c1\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") "
Jul 23 22:37:15.261723 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.261556    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-home\") pod \"574e2f38-f339-4519-a89f-dbfa28d408c1\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") "
Jul 23 22:37:15.261723 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.261600    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-tmp-dir\") pod \"574e2f38-f339-4519-a89f-dbfa28d408c1\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") "
Jul 23 22:37:15.261723 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.261614    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-dshm\") pod \"574e2f38-f339-4519-a89f-dbfa28d408c1\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") "
Jul 23 22:37:15.261723 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.261638    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/574e2f38-f339-4519-a89f-dbfa28d408c1-tls-certs\") pod \"574e2f38-f339-4519-a89f-dbfa28d408c1\" (UID: \"574e2f38-f339-4519-a89f-dbfa28d408c1\") "
Jul 23 22:37:15.262015 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.261905    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "574e2f38-f339-4519-a89f-dbfa28d408c1" (UID: "574e2f38-f339-4519-a89f-dbfa28d408c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:37:15.262015 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.261911    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "574e2f38-f339-4519-a89f-dbfa28d408c1" (UID: "574e2f38-f339-4519-a89f-dbfa28d408c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:37:15.262015 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.261934    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-home" (OuterVolumeSpecName: "home") pod "574e2f38-f339-4519-a89f-dbfa28d408c1" (UID: "574e2f38-f339-4519-a89f-dbfa28d408c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:37:15.263788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.263760    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/574e2f38-f339-4519-a89f-dbfa28d408c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "574e2f38-f339-4519-a89f-dbfa28d408c1" (UID: "574e2f38-f339-4519-a89f-dbfa28d408c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:37:15.263788 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.263780    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-dshm" (OuterVolumeSpecName: "dshm") pod "574e2f38-f339-4519-a89f-dbfa28d408c1" (UID: "574e2f38-f339-4519-a89f-dbfa28d408c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:37:15.264008 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.263896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574e2f38-f339-4519-a89f-dbfa28d408c1-kube-api-access-6vkvh" (OuterVolumeSpecName: "kube-api-access-6vkvh") pod "574e2f38-f339-4519-a89f-dbfa28d408c1" (UID: "574e2f38-f339-4519-a89f-dbfa28d408c1"). InnerVolumeSpecName "kube-api-access-6vkvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:37:15.363167 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.363077    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vkvh\" (UniqueName: \"kubernetes.io/projected/574e2f38-f339-4519-a89f-dbfa28d408c1-kube-api-access-6vkvh\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:15.363167 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.363110    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:15.363167 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.363120    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:15.363167 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.363150    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:15.363167 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.363158    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/574e2f38-f339-4519-a89f-dbfa28d408c1-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:15.363167 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.363165    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/574e2f38-f339-4519-a89f-dbfa28d408c1-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:37:15.743787 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.743747    2579 generic.go:356] "Generic (PLEG): container finished" podID="574e2f38-f339-4519-a89f-dbfa28d408c1" containerID="4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044" exitCode=0
Jul 23 22:37:15.743787 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.743789    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"
Jul 23 22:37:15.744052 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.743834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl" event={"ID":"574e2f38-f339-4519-a89f-dbfa28d408c1","Type":"ContainerDied","Data":"4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044"}
Jul 23 22:37:15.744052 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.743903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl" event={"ID":"574e2f38-f339-4519-a89f-dbfa28d408c1","Type":"ContainerDied","Data":"9e917316604d03f762aa10122c978563dc701d417a3fe632039702e51a2f0806"}
Jul 23 22:37:15.744052 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.743927    2579 scope.go:117] "RemoveContainer" containerID="4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044"
Jul 23 22:37:15.752755 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.752737    2579 scope.go:117] "RemoveContainer" containerID="4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044"
Jul 23 22:37:15.753119 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:37:15.753099    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044\": container with ID starting with 4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044 not found: ID does not exist" containerID="4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044"
Jul 23 22:37:15.753192 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.753130    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044"} err="failed to get container status \"4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044\": rpc error: code = NotFound desc = could not find container \"4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044\": container with ID starting with 4896314b16caad3fb71e9ff70b77653b69c08126afd4595ff38bc58bdc841044 not found: ID does not exist"
Jul 23 22:37:15.764539 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.764513    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"]
Jul 23 22:37:15.768309 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:15.768285    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/sub-test-model-091ba5-kserve-fbffcb747-77gxl"]
Jul 23 22:37:17.112516 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:17.112477    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="574e2f38-f339-4519-a89f-dbfa28d408c1" path="/var/lib/kubelet/pods/574e2f38-f339-4519-a89f-dbfa28d408c1/volumes"
Jul 23 22:37:21.615796 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.615765    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"]
Jul 23 22:37:21.618237 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.616146    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c446f1a-ccde-4b7a-97f6-73794a12233d" containerName="main"
Jul 23 22:37:21.618237 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.616172    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c446f1a-ccde-4b7a-97f6-73794a12233d" containerName="main"
Jul 23 22:37:21.618237 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.616194    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="574e2f38-f339-4519-a89f-dbfa28d408c1" containerName="main"
Jul 23 22:37:21.618237 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.616200    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="574e2f38-f339-4519-a89f-dbfa28d408c1" containerName="main"
Jul 23 22:37:21.618237 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.616249    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="574e2f38-f339-4519-a89f-dbfa28d408c1" containerName="main"
Jul 23 22:37:21.618237 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.616265    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c446f1a-ccde-4b7a-97f6-73794a12233d" containerName="main"
Jul 23 22:37:21.619192 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.619169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.620923 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.620892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tls-certs\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.621062 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.620940    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tmp-dir\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.621062 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.621039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-dshm\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.621181 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.621087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-home\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.621181 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.621166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj6pw\" (UniqueName: \"kubernetes.io/projected/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-kube-api-access-tj6pw\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.621286 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.621215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-model-cache\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.622644 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.622618    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a1850d\"/\"kube-root-ca.crt\""
Jul 23 22:37:21.622762 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.622647    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a1850d\"/\"openshift-service-ca.crt\""
Jul 23 22:37:21.622762 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.622677    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a1850d\"/\"rate-test-model-a1850d-kserve-self-signed-certs\""
Jul 23 22:37:21.622762 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.622618    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a1850d\"/\"default-dockercfg-zdrb5\""
Jul 23 22:37:21.630206 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.630177    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"]
Jul 23 22:37:21.654228 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.654190    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:37:21.721962 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.721922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tls-certs\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.721962 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.721963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tmp-dir\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.722234 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.721989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-dshm\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.722234 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.722011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-home\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.722234 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.722051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj6pw\" (UniqueName: \"kubernetes.io/projected/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-kube-api-access-tj6pw\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.722234 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.722078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-model-cache\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.722510 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.722389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tmp-dir\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.722510 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.722463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-home\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.722510 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.722490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-model-cache\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.724497 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.724474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-dshm\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.724832 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.724814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tls-certs\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.730228 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.730202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj6pw\" (UniqueName: \"kubernetes.io/projected/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-kube-api-access-tj6pw\") pod \"rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") " pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:21.933679 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:21.933582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:22.094091 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:22.094063    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"]
Jul 23 22:37:22.095877 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:37:22.095823    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdbe64cc_a666_4df8_b0a2_a227d71f0a0c.slice/crio-a13201dbc0df4f465993a1ec1a4a0ad9a8ebf30cdc124a2fbd70ed28fedbf41c WatchSource:0}: Error finding container a13201dbc0df4f465993a1ec1a4a0ad9a8ebf30cdc124a2fbd70ed28fedbf41c: Status 404 returned error can't find the container with id a13201dbc0df4f465993a1ec1a4a0ad9a8ebf30cdc124a2fbd70ed28fedbf41c
Jul 23 22:37:22.780456 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:22.780427    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76" event={"ID":"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c","Type":"ContainerStarted","Data":"3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc"}
Jul 23 22:37:22.780854 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:22.780466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76" event={"ID":"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c","Type":"ContainerStarted","Data":"a13201dbc0df4f465993a1ec1a4a0ad9a8ebf30cdc124a2fbd70ed28fedbf41c"}
Jul 23 22:37:22.800041 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:22.799988    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76" podStartSLOduration=1.79997126 podStartE2EDuration="1.79997126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:37:22.79923326 +0000 UTC m=+2738.189625876" watchObservedRunningTime="2026-07-23 22:37:22.79997126 +0000 UTC m=+2738.190363876"
Jul 23 22:37:30.793471 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:30.793434    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:37:30.894077 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:30.894038    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:37:31.934019 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:31.933982    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:31.934424 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:31.934045    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:31.946645 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:31.946613    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:32.842472 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:32.842441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:37:39.509276 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.509242    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"]
Jul 23 22:37:39.512770 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.512754    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.516991 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.516962    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"kube-root-ca.crt\""
Jul 23 22:37:39.517148 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.517018    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"default-dockercfg-gjrm9\""
Jul 23 22:37:39.517148 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.516968    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"rate-test-model-091ba5-kserve-self-signed-certs\""
Jul 23 22:37:39.517148 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.516968    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-091ba5\"/\"openshift-service-ca.crt\""
Jul 23 22:37:39.522728 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.522701    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"]
Jul 23 22:37:39.542781 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.542745    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:37:39.563936 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.563902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-home\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.564157 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.563958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tmp-dir\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.564157 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.563983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-dshm\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.564157 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.564054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-model-cache\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.564157 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.564107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tls-certs\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.564329 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.564172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq4kf\" (UniqueName: \"kubernetes.io/projected/1624774e-2c8a-4718-8e8f-b07e5117fbe9-kube-api-access-hq4kf\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.665644 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.665575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-home\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.665644 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.665639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tmp-dir\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.666035 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.665664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-dshm\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.666035 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.665689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-model-cache\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.666035 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.665712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tls-certs\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.666188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.666068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq4kf\" (UniqueName: \"kubernetes.io/projected/1624774e-2c8a-4718-8e8f-b07e5117fbe9-kube-api-access-hq4kf\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.666188 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.666137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-model-cache\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.666291 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.666184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tmp-dir\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.666291 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.666241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-home\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.668692 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.668662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-dshm\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.669006 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.668983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tls-certs\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.676538 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.676477    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq4kf\" (UniqueName: \"kubernetes.io/projected/1624774e-2c8a-4718-8e8f-b07e5117fbe9-kube-api-access-hq4kf\") pod \"rate-test-model-091ba5-kserve-76f94784cf-xfdwh\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") " pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.824105 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.824009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:39.968544 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.968496    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"]
Jul 23 22:37:39.971610 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:37:39.971586    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1624774e_2c8a_4718_8e8f_b07e5117fbe9.slice/crio-e82421ab744ede7da02dc33870044c446f3a97696e2b4a15c16876a84cc46590 WatchSource:0}: Error finding container e82421ab744ede7da02dc33870044c446f3a97696e2b4a15c16876a84cc46590: Status 404 returned error can't find the container with id e82421ab744ede7da02dc33870044c446f3a97696e2b4a15c16876a84cc46590
Jul 23 22:37:39.977526 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:39.977497    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:37:40.036900 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:40.036839    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:37:40.867924 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:40.867876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh" event={"ID":"1624774e-2c8a-4718-8e8f-b07e5117fbe9","Type":"ContainerStarted","Data":"47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b"}
Jul 23 22:37:40.867924 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:40.867927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh" event={"ID":"1624774e-2c8a-4718-8e8f-b07e5117fbe9","Type":"ContainerStarted","Data":"e82421ab744ede7da02dc33870044c446f3a97696e2b4a15c16876a84cc46590"}
Jul 23 22:37:40.886016 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:40.885963    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh" podStartSLOduration=1.885944719 podStartE2EDuration="1.885944719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:37:40.885445419 +0000 UTC m=+2756.275838038" watchObservedRunningTime="2026-07-23 22:37:40.885944719 +0000 UTC m=+2756.276337335"
Jul 23 22:37:45.442888 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:45.442827    2579 scope.go:117] "RemoveContainer" containerID="fd7520265aa6d37bf04b207d4c6002e326a56437614820fe534d14db5f949a10"
Jul 23 22:37:46.701808 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:46.701750    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:37:46.762350 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:46.762316    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:37:49.824683 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:49.824641    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:49.825121 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:49.824733    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:49.837514 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:49.837482    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:37:49.922381 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:37:49.922353    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:38:32.433275 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:32.433235    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"]
Jul 23 22:38:32.444450 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:32.444406    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:38:33.425050 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:33.425011    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"]
Jul 23 22:38:33.444792 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:33.444745    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:38:47.454001 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.453950    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76" podUID="fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" containerName="main" containerID="cri-o://3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc" gracePeriod=45
Jul 23 22:38:47.706886 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.706793    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:38:47.782921 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.782882    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-dshm\") pod \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") "
Jul 23 22:38:47.783120 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.782938    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-model-cache\") pod \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") "
Jul 23 22:38:47.783120 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.782969    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tmp-dir\") pod \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") "
Jul 23 22:38:47.783120 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.783017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tls-certs\") pod \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") "
Jul 23 22:38:47.783120 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.783045    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj6pw\" (UniqueName: \"kubernetes.io/projected/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-kube-api-access-tj6pw\") pod \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") "
Jul 23 22:38:47.783337 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.783128    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-home\") pod \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\" (UID: \"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c\") "
Jul 23 22:38:47.783337 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.783250    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" (UID: "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:38:47.783337 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.783267    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" (UID: "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:38:47.783512 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.783476    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:47.783512 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.783484    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-home" (OuterVolumeSpecName: "home") pod "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" (UID: "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:38:47.783593 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.783520    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:47.785201 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.785174    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-dshm" (OuterVolumeSpecName: "dshm") pod "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" (UID: "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:38:47.785465 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.785439    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-kube-api-access-tj6pw" (OuterVolumeSpecName: "kube-api-access-tj6pw") pod "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" (UID: "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c"). InnerVolumeSpecName "kube-api-access-tj6pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:38:47.785592 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.785562    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" (UID: "fdbe64cc-a666-4df8-b0a2-a227d71f0a0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:38:47.884152 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.884104    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:47.884152 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.884143    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:47.884152 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.884155    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:47.884401 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:47.884171    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj6pw\" (UniqueName: \"kubernetes.io/projected/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c-kube-api-access-tj6pw\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:48.158938 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.158821    2579 generic.go:356] "Generic (PLEG): container finished" podID="fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" containerID="3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc" exitCode=0
Jul 23 22:38:48.158938 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.158856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76" event={"ID":"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c","Type":"ContainerDied","Data":"3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc"}
Jul 23 22:38:48.158938 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.158905    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"
Jul 23 22:38:48.158938 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.158921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76" event={"ID":"fdbe64cc-a666-4df8-b0a2-a227d71f0a0c","Type":"ContainerDied","Data":"a13201dbc0df4f465993a1ec1a4a0ad9a8ebf30cdc124a2fbd70ed28fedbf41c"}
Jul 23 22:38:48.158938 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.158934    2579 scope.go:117] "RemoveContainer" containerID="3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc"
Jul 23 22:38:48.167720 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.167698    2579 scope.go:117] "RemoveContainer" containerID="3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc"
Jul 23 22:38:48.168070 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:38:48.168041    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc\": container with ID starting with 3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc not found: ID does not exist" containerID="3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc"
Jul 23 22:38:48.168152 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.168081    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc"} err="failed to get container status \"3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc\": rpc error: code = NotFound desc = could not find container \"3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc\": container with ID starting with 3f75759af20d159bfb14375ca8a82d18b18f5315d13a54c9ec9304a28fd0f8dc not found: ID does not exist"
Jul 23 22:38:48.178564 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.178532    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"]
Jul 23 22:38:48.182295 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.182269    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a1850d/rate-test-model-a1850d-kserve-6dc8c58ccf-ffv76"]
Jul 23 22:38:48.446973 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.446836    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh" podUID="1624774e-2c8a-4718-8e8f-b07e5117fbe9" containerName="main" containerID="cri-o://47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b" gracePeriod=45
Jul 23 22:38:48.696337 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.696307    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:38:48.793071 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793038    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-model-cache\") pod \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") "
Jul 23 22:38:48.793281 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793109    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-dshm\") pod \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") "
Jul 23 22:38:48.793281 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-home\") pod \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") "
Jul 23 22:38:48.793281 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793169    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tmp-dir\") pod \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") "
Jul 23 22:38:48.793281 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793201    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq4kf\" (UniqueName: \"kubernetes.io/projected/1624774e-2c8a-4718-8e8f-b07e5117fbe9-kube-api-access-hq4kf\") pod \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") "
Jul 23 22:38:48.793281 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tls-certs\") pod \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\" (UID: \"1624774e-2c8a-4718-8e8f-b07e5117fbe9\") "
Jul 23 22:38:48.793571 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793379    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-model-cache" (OuterVolumeSpecName: "model-cache") pod "1624774e-2c8a-4718-8e8f-b07e5117fbe9" (UID: "1624774e-2c8a-4718-8e8f-b07e5117fbe9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:38:48.793571 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793450    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-home" (OuterVolumeSpecName: "home") pod "1624774e-2c8a-4718-8e8f-b07e5117fbe9" (UID: "1624774e-2c8a-4718-8e8f-b07e5117fbe9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:38:48.793571 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793468    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1624774e-2c8a-4718-8e8f-b07e5117fbe9" (UID: "1624774e-2c8a-4718-8e8f-b07e5117fbe9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:38:48.793571 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793553    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-home\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:48.793571 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793570    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tmp-dir\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:48.793742 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.793584    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-model-cache\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:48.795424 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.795400    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-dshm" (OuterVolumeSpecName: "dshm") pod "1624774e-2c8a-4718-8e8f-b07e5117fbe9" (UID: "1624774e-2c8a-4718-8e8f-b07e5117fbe9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:38:48.795532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.795444    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1624774e-2c8a-4718-8e8f-b07e5117fbe9-kube-api-access-hq4kf" (OuterVolumeSpecName: "kube-api-access-hq4kf") pod "1624774e-2c8a-4718-8e8f-b07e5117fbe9" (UID: "1624774e-2c8a-4718-8e8f-b07e5117fbe9"). InnerVolumeSpecName "kube-api-access-hq4kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:38:48.795532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.795493    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1624774e-2c8a-4718-8e8f-b07e5117fbe9" (UID: "1624774e-2c8a-4718-8e8f-b07e5117fbe9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:38:48.894127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.894082    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1624774e-2c8a-4718-8e8f-b07e5117fbe9-dshm\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:48.894127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.894118    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq4kf\" (UniqueName: \"kubernetes.io/projected/1624774e-2c8a-4718-8e8f-b07e5117fbe9-kube-api-access-hq4kf\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:48.894127 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:48.894134    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1624774e-2c8a-4718-8e8f-b07e5117fbe9-tls-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:49.112461 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.112378    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" path="/var/lib/kubelet/pods/fdbe64cc-a666-4df8-b0a2-a227d71f0a0c/volumes"
Jul 23 22:38:49.165660 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.165623    2579 generic.go:356] "Generic (PLEG): container finished" podID="1624774e-2c8a-4718-8e8f-b07e5117fbe9" containerID="47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b" exitCode=0
Jul 23 22:38:49.165660 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.165663    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"
Jul 23 22:38:49.165908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.165705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh" event={"ID":"1624774e-2c8a-4718-8e8f-b07e5117fbe9","Type":"ContainerDied","Data":"47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b"}
Jul 23 22:38:49.165908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.165749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh" event={"ID":"1624774e-2c8a-4718-8e8f-b07e5117fbe9","Type":"ContainerDied","Data":"e82421ab744ede7da02dc33870044c446f3a97696e2b4a15c16876a84cc46590"}
Jul 23 22:38:49.165908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.165766    2579 scope.go:117] "RemoveContainer" containerID="47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b"
Jul 23 22:38:49.175020 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.174996    2579 scope.go:117] "RemoveContainer" containerID="47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b"
Jul 23 22:38:49.175325 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:38:49.175307    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b\": container with ID starting with 47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b not found: ID does not exist" containerID="47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b"
Jul 23 22:38:49.175386 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.175332    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b"} err="failed to get container status \"47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b\": rpc error: code = NotFound desc = could not find container \"47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b\": container with ID starting with 47b433a23e8c108a91e6caed33da7075eacebebbe3a0ef8d414d1c592f75553b not found: ID does not exist"
Jul 23 22:38:49.181474 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.181444    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"]
Jul 23 22:38:49.184728 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.184703    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-091ba5/rate-test-model-091ba5-kserve-76f94784cf-xfdwh"]
Jul 23 22:38:49.864851 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.864817    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7721-139888552786080"]
Jul 23 22:38:49.865264 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.865177    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1624774e-2c8a-4718-8e8f-b07e5117fbe9" containerName="main"
Jul 23 22:38:49.865264 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.865188    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1624774e-2c8a-4718-8e8f-b07e5117fbe9" containerName="main"
Jul 23 22:38:49.865264 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.865208    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" containerName="main"
Jul 23 22:38:49.865264 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.865213    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" containerName="main"
Jul 23 22:38:49.865264 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.865262    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdbe64cc-a666-4df8-b0a2-a227d71f0a0c" containerName="main"
Jul 23 22:38:49.865425 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.865272    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1624774e-2c8a-4718-8e8f-b07e5117fbe9" containerName="main"
Jul 23 22:38:49.869696 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.869677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7721-139888552786080"
Jul 23 22:38:49.872094 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.872069    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bqb75\""
Jul 23 22:38:49.872575 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:49.872548    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7721-139888552786080"]
Jul 23 22:38:50.005270 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:50.005234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnmrt\" (UniqueName: \"kubernetes.io/projected/d6f4363e-2e6c-4b07-a9a6-2419a09d0142-kube-api-access-tnmrt\") pod \"test-curl-7721-139888552786080\" (UID: \"d6f4363e-2e6c-4b07-a9a6-2419a09d0142\") " pod="opendatahub/test-curl-7721-139888552786080"
Jul 23 22:38:50.106689 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:50.106646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnmrt\" (UniqueName: \"kubernetes.io/projected/d6f4363e-2e6c-4b07-a9a6-2419a09d0142-kube-api-access-tnmrt\") pod \"test-curl-7721-139888552786080\" (UID: \"d6f4363e-2e6c-4b07-a9a6-2419a09d0142\") " pod="opendatahub/test-curl-7721-139888552786080"
Jul 23 22:38:50.114678 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:50.114639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnmrt\" (UniqueName: \"kubernetes.io/projected/d6f4363e-2e6c-4b07-a9a6-2419a09d0142-kube-api-access-tnmrt\") pod \"test-curl-7721-139888552786080\" (UID: \"d6f4363e-2e6c-4b07-a9a6-2419a09d0142\") " pod="opendatahub/test-curl-7721-139888552786080"
Jul 23 22:38:50.180505 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:50.180422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7721-139888552786080"
Jul 23 22:38:50.316570 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:50.316526    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7721-139888552786080"]
Jul 23 22:38:50.321611 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:38:50.321580    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6f4363e_2e6c_4b07_a9a6_2419a09d0142.slice/crio-8747a4ad50d74710e0fb4755836e79871960fdcc52c7bf1e732d17631cc0633d WatchSource:0}: Error finding container 8747a4ad50d74710e0fb4755836e79871960fdcc52c7bf1e732d17631cc0633d: Status 404 returned error can't find the container with id 8747a4ad50d74710e0fb4755836e79871960fdcc52c7bf1e732d17631cc0633d
Jul 23 22:38:50.323058 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:50.323036    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 22:38:51.111648 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:51.111614    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1624774e-2c8a-4718-8e8f-b07e5117fbe9" path="/var/lib/kubelet/pods/1624774e-2c8a-4718-8e8f-b07e5117fbe9/volumes"
Jul 23 22:38:51.177167 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:51.177137    2579 generic.go:356] "Generic (PLEG): container finished" podID="d6f4363e-2e6c-4b07-a9a6-2419a09d0142" containerID="ab826d6799df3ad0aea1b1a44ca3c7a3bdc41ce282f05250c78889f14d1ecd9b" exitCode=0
Jul 23 22:38:51.177356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:51.177192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7721-139888552786080" event={"ID":"d6f4363e-2e6c-4b07-a9a6-2419a09d0142","Type":"ContainerDied","Data":"ab826d6799df3ad0aea1b1a44ca3c7a3bdc41ce282f05250c78889f14d1ecd9b"}
Jul 23 22:38:51.177356 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:51.177217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7721-139888552786080" event={"ID":"d6f4363e-2e6c-4b07-a9a6-2419a09d0142","Type":"ContainerStarted","Data":"8747a4ad50d74710e0fb4755836e79871960fdcc52c7bf1e732d17631cc0633d"}
Jul 23 22:38:52.314665 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:52.314640    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7721-139888552786080"
Jul 23 22:38:52.336252 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:52.336223    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7721-139888552786080_d6f4363e-2e6c-4b07-a9a6-2419a09d0142/test-curl-7721-139888552786080/0.log"
Jul 23 22:38:52.361405 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:52.361363    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7721-139888552786080"]
Jul 23 22:38:52.367627 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:52.367597    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7721-139888552786080"]
Jul 23 22:38:52.428668 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:52.428620    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnmrt\" (UniqueName: \"kubernetes.io/projected/d6f4363e-2e6c-4b07-a9a6-2419a09d0142-kube-api-access-tnmrt\") pod \"d6f4363e-2e6c-4b07-a9a6-2419a09d0142\" (UID: \"d6f4363e-2e6c-4b07-a9a6-2419a09d0142\") "
Jul 23 22:38:52.431007 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:52.430980    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f4363e-2e6c-4b07-a9a6-2419a09d0142-kube-api-access-tnmrt" (OuterVolumeSpecName: "kube-api-access-tnmrt") pod "d6f4363e-2e6c-4b07-a9a6-2419a09d0142" (UID: "d6f4363e-2e6c-4b07-a9a6-2419a09d0142"). InnerVolumeSpecName "kube-api-access-tnmrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:38:52.529728 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:52.529687    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnmrt\" (UniqueName: \"kubernetes.io/projected/d6f4363e-2e6c-4b07-a9a6-2419a09d0142-kube-api-access-tnmrt\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:38:53.112180 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:53.112138    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6f4363e-2e6c-4b07-a9a6-2419a09d0142" path="/var/lib/kubelet/pods/d6f4363e-2e6c-4b07-a9a6-2419a09d0142/volumes"
Jul 23 22:38:53.137921 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:53.137836    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:38:53.186879 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:53.186836    2579 scope.go:117] "RemoveContainer" containerID="ab826d6799df3ad0aea1b1a44ca3c7a3bdc41ce282f05250c78889f14d1ecd9b"
Jul 23 22:38:53.187055 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:53.186846    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7721-139888552786080"
Jul 23 22:38:53.237488 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:38:53.237452    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:02.988002 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:02.987968    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"]
Jul 23 22:39:02.988486 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:02.988216    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9" podUID="20a7bbce-775b-47ea-9b08-fa9e7018af7b" containerName="maas-api" containerID="cri-o://0758c8d945b07d346c9ef6b4f47cb088318f2fcab09aaff8e485a454972bdbf2" gracePeriod=30
Jul 23 22:39:03.012086 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.012044    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:03.234464 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.234434    2579 generic.go:356] "Generic (PLEG): container finished" podID="20a7bbce-775b-47ea-9b08-fa9e7018af7b" containerID="0758c8d945b07d346c9ef6b4f47cb088318f2fcab09aaff8e485a454972bdbf2" exitCode=0
Jul 23 22:39:03.234615 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.234499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9" event={"ID":"20a7bbce-775b-47ea-9b08-fa9e7018af7b","Type":"ContainerDied","Data":"0758c8d945b07d346c9ef6b4f47cb088318f2fcab09aaff8e485a454972bdbf2"}
Jul 23 22:39:03.234615 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.234559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9" event={"ID":"20a7bbce-775b-47ea-9b08-fa9e7018af7b","Type":"ContainerDied","Data":"aa494138d1217e2e0dc3e8e365c316bae0e5798d73c4d4ff97c0d7a74b4004ae"}
Jul 23 22:39:03.234615 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.234582    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa494138d1217e2e0dc3e8e365c316bae0e5798d73c4d4ff97c0d7a74b4004ae"
Jul 23 22:39:03.237917 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.237897    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:39:03.425283 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.425191    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a7bbce-775b-47ea-9b08-fa9e7018af7b-openshift-service-ca\") pod \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") "
Jul 23 22:39:03.425283 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.425246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a7bbce-775b-47ea-9b08-fa9e7018af7b-maas-api-tls\") pod \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") "
Jul 23 22:39:03.425508 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.425308    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2895z\" (UniqueName: \"kubernetes.io/projected/20a7bbce-775b-47ea-9b08-fa9e7018af7b-kube-api-access-2895z\") pod \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\" (UID: \"20a7bbce-775b-47ea-9b08-fa9e7018af7b\") "
Jul 23 22:39:03.425567 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.425541    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a7bbce-775b-47ea-9b08-fa9e7018af7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "20a7bbce-775b-47ea-9b08-fa9e7018af7b" (UID: "20a7bbce-775b-47ea-9b08-fa9e7018af7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:39:03.427530 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.427502    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a7bbce-775b-47ea-9b08-fa9e7018af7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "20a7bbce-775b-47ea-9b08-fa9e7018af7b" (UID: "20a7bbce-775b-47ea-9b08-fa9e7018af7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:39:03.427530 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.427510    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a7bbce-775b-47ea-9b08-fa9e7018af7b-kube-api-access-2895z" (OuterVolumeSpecName: "kube-api-access-2895z") pod "20a7bbce-775b-47ea-9b08-fa9e7018af7b" (UID: "20a7bbce-775b-47ea-9b08-fa9e7018af7b"). InnerVolumeSpecName "kube-api-access-2895z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:39:03.526329 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.526290    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20a7bbce-775b-47ea-9b08-fa9e7018af7b-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:03.526329 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.526322    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20a7bbce-775b-47ea-9b08-fa9e7018af7b-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:03.526329 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:03.526332    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2895z\" (UniqueName: \"kubernetes.io/projected/20a7bbce-775b-47ea-9b08-fa9e7018af7b-kube-api-access-2895z\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:04.238287 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:04.238255    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"
Jul 23 22:39:04.258487 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:04.258459    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"]
Jul 23 22:39:04.262766 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:04.262742    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a1850d-97d495ff6-r2qs9"]
Jul 23 22:39:05.111098 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:05.111067    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a7bbce-775b-47ea-9b08-fa9e7018af7b" path="/var/lib/kubelet/pods/20a7bbce-775b-47ea-9b08-fa9e7018af7b/volumes"
Jul 23 22:39:07.649005 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:07.648972    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:08.074038 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:08.074003    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:14.503614 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:14.503579    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:14.568694 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:14.568660    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:14.773779 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:14.773681    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"]
Jul 23 22:39:14.774040 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:14.773997    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy" containerID="cri-o://fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232" gracePeriod=30
Jul 23 22:39:14.790837 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:14.790806    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:15.029913 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.029815    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:15.155983 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.155941    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"]
Jul 23 22:39:15.156275 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.156262    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a7bbce-775b-47ea-9b08-fa9e7018af7b" containerName="maas-api"
Jul 23 22:39:15.156275 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.156277    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a7bbce-775b-47ea-9b08-fa9e7018af7b" containerName="maas-api"
Jul 23 22:39:15.156379 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.156286    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f4363e-2e6c-4b07-a9a6-2419a09d0142" containerName="test-curl-7721-139888552786080"
Jul 23 22:39:15.156379 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.156292    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f4363e-2e6c-4b07-a9a6-2419a09d0142" containerName="test-curl-7721-139888552786080"
Jul 23 22:39:15.156379 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.156347    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f4363e-2e6c-4b07-a9a6-2419a09d0142" containerName="test-curl-7721-139888552786080"
Jul 23 22:39:15.156379 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.156358    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a7bbce-775b-47ea-9b08-fa9e7018af7b" containerName="maas-api"
Jul 23 22:39:15.159467 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.159450    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"
Jul 23 22:39:15.161818 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.161799    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lq4mm\""
Jul 23 22:39:15.167449 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.167423    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"]
Jul 23 22:39:15.225770 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.225730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb33a4c-b972-4818-a151-d50b9eb55b76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt\" (UID: \"bfb33a4c-b972-4818-a151-d50b9eb55b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"
Jul 23 22:39:15.327144 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.327044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb33a4c-b972-4818-a151-d50b9eb55b76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt\" (UID: \"bfb33a4c-b972-4818-a151-d50b9eb55b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"
Jul 23 22:39:15.327779 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.327751    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb33a4c-b972-4818-a151-d50b9eb55b76-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt\" (UID: \"bfb33a4c-b972-4818-a151-d50b9eb55b76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"
Jul 23 22:39:15.470974 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.470930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"
Jul 23 22:39:15.609088 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.608917    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"]
Jul 23 22:39:15.611719 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:39:15.611685    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb33a4c_b972_4818_a151_d50b9eb55b76.slice/crio-81b539be9a220dab3c4b4b967e00f3625e2c05e53f4d4b102a942602861e8141 WatchSource:0}: Error finding container 81b539be9a220dab3c4b4b967e00f3625e2c05e53f4d4b102a942602861e8141: Status 404 returned error can't find the container with id 81b539be9a220dab3c4b4b967e00f3625e2c05e53f4d4b102a942602861e8141
Jul 23 22:39:15.690468 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.690441    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:15.791377 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:15.791324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:16.293751 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:16.293719    2579 generic.go:356] "Generic (PLEG): container finished" podID="bfb33a4c-b972-4818-a151-d50b9eb55b76" containerID="45a70cc55640839accf9a19a1ac9753bfef63d585f6da3277475db6ed8714e4d" exitCode=0
Jul 23 22:39:16.293926 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:16.293803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt" event={"ID":"bfb33a4c-b972-4818-a151-d50b9eb55b76","Type":"ContainerDied","Data":"45a70cc55640839accf9a19a1ac9753bfef63d585f6da3277475db6ed8714e4d"}
Jul 23 22:39:16.293926 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:16.293884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt" event={"ID":"bfb33a4c-b972-4818-a151-d50b9eb55b76","Type":"ContainerStarted","Data":"81b539be9a220dab3c4b4b967e00f3625e2c05e53f4d4b102a942602861e8141"}
Jul 23 22:39:17.432781 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:17.432749    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"
Jul 23 22:39:17.445617 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:17.445584    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb33a4c-b972-4818-a151-d50b9eb55b76-maas-api-service-ca\") pod \"bfb33a4c-b972-4818-a151-d50b9eb55b76\" (UID: \"bfb33a4c-b972-4818-a151-d50b9eb55b76\") "
Jul 23 22:39:17.445950 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:17.445927    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb33a4c-b972-4818-a151-d50b9eb55b76-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bfb33a4c-b972-4818-a151-d50b9eb55b76" (UID: "bfb33a4c-b972-4818-a151-d50b9eb55b76"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:39:17.546586 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:17.546548    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb33a4c-b972-4818-a151-d50b9eb55b76-maas-api-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:18.303929 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:18.303895    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"
Jul 23 22:39:18.304117 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:18.303900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt" event={"ID":"bfb33a4c-b972-4818-a151-d50b9eb55b76","Type":"ContainerDied","Data":"81b539be9a220dab3c4b4b967e00f3625e2c05e53f4d4b102a942602861e8141"}
Jul 23 22:39:18.304117 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:18.304021    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81b539be9a220dab3c4b4b967e00f3625e2c05e53f4d4b102a942602861e8141"
Jul 23 22:39:20.027048 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.027021    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:39:20.067574 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.067479    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-credential-socket\") pod \"00a473ba-7800-4e53-baca-43db45064258\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") "
Jul 23 22:39:20.067574 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.067512    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-data\") pod \"00a473ba-7800-4e53-baca-43db45064258\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") "
Jul 23 22:39:20.067574 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.067537    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a473ba-7800-4e53-baca-43db45064258-istiod-ca-cert\") pod \"00a473ba-7800-4e53-baca-43db45064258\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") "
Jul 23 22:39:20.067574 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.067564    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a473ba-7800-4e53-baca-43db45064258-istio-podinfo\") pod \"00a473ba-7800-4e53-baca-43db45064258\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") "
Jul 23 22:39:20.068030 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.067764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnpt6\" (UniqueName: \"kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-kube-api-access-xnpt6\") pod \"00a473ba-7800-4e53-baca-43db45064258\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") "
Jul 23 22:39:20.068030 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.067819    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-istio-token\") pod \"00a473ba-7800-4e53-baca-43db45064258\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") "
Jul 23 22:39:20.068030 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.067848    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-envoy\") pod \"00a473ba-7800-4e53-baca-43db45064258\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") "
Jul 23 22:39:20.068030 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.067826    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "00a473ba-7800-4e53-baca-43db45064258" (UID: "00a473ba-7800-4e53-baca-43db45064258"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:39:20.068030 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.067957    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-certs\") pod \"00a473ba-7800-4e53-baca-43db45064258\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") "
Jul 23 22:39:20.068030 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.067998    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-socket\") pod \"00a473ba-7800-4e53-baca-43db45064258\" (UID: \"00a473ba-7800-4e53-baca-43db45064258\") "
Jul 23 22:39:20.068030 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.068012    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a473ba-7800-4e53-baca-43db45064258-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "00a473ba-7800-4e53-baca-43db45064258" (UID: "00a473ba-7800-4e53-baca-43db45064258"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:39:20.068401 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.068293    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-credential-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:20.068401 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.068311    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a473ba-7800-4e53-baca-43db45064258-istiod-ca-cert\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:20.068591 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.068419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "00a473ba-7800-4e53-baca-43db45064258" (UID: "00a473ba-7800-4e53-baca-43db45064258"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:39:20.068728 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.068587    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "00a473ba-7800-4e53-baca-43db45064258" (UID: "00a473ba-7800-4e53-baca-43db45064258"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:39:20.068816 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.068787    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-data" (OuterVolumeSpecName: "istio-data") pod "00a473ba-7800-4e53-baca-43db45064258" (UID: "00a473ba-7800-4e53-baca-43db45064258"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:39:20.070203 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.070177    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00a473ba-7800-4e53-baca-43db45064258-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "00a473ba-7800-4e53-baca-43db45064258" (UID: "00a473ba-7800-4e53-baca-43db45064258"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 23 22:39:20.070341 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.070205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-istio-token" (OuterVolumeSpecName: "istio-token") pod "00a473ba-7800-4e53-baca-43db45064258" (UID: "00a473ba-7800-4e53-baca-43db45064258"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:39:20.070721 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.070690    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "00a473ba-7800-4e53-baca-43db45064258" (UID: "00a473ba-7800-4e53-baca-43db45064258"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 22:39:20.070721 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.070707    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-kube-api-access-xnpt6" (OuterVolumeSpecName: "kube-api-access-xnpt6") pod "00a473ba-7800-4e53-baca-43db45064258" (UID: "00a473ba-7800-4e53-baca-43db45064258"). InnerVolumeSpecName "kube-api-access-xnpt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:39:20.169394 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.169355    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-data\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:20.169394 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.169389    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a473ba-7800-4e53-baca-43db45064258-istio-podinfo\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:20.169394 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.169398    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnpt6\" (UniqueName: \"kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-kube-api-access-xnpt6\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:20.169639 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.169409    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a473ba-7800-4e53-baca-43db45064258-istio-token\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:20.169639 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.169423    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-istio-envoy\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:20.169639 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.169434    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-certs\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:20.169639 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.169443    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a473ba-7800-4e53-baca-43db45064258-workload-socket\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:20.314401 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.314368    2579 generic.go:356] "Generic (PLEG): container finished" podID="00a473ba-7800-4e53-baca-43db45064258" containerID="fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232" exitCode=0
Jul 23 22:39:20.314590 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.314426    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"
Jul 23 22:39:20.314590 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.314468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" event={"ID":"00a473ba-7800-4e53-baca-43db45064258","Type":"ContainerDied","Data":"fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232"}
Jul 23 22:39:20.314590 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.314525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2" event={"ID":"00a473ba-7800-4e53-baca-43db45064258","Type":"ContainerDied","Data":"d16ec4d8f207c938e0d7dbd4ce0391990c949d18250cb083cfb33fa78af1139b"}
Jul 23 22:39:20.314590 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.314551    2579 scope.go:117] "RemoveContainer" containerID="fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232"
Jul 23 22:39:20.323837 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.323819    2579 scope.go:117] "RemoveContainer" containerID="997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184"
Jul 23 22:39:20.331672 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.331651    2579 scope.go:117] "RemoveContainer" containerID="fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232"
Jul 23 22:39:20.331993 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:39:20.331973    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232\": container with ID starting with fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232 not found: ID does not exist" containerID="fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232"
Jul 23 22:39:20.332043 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.332003    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232"} err="failed to get container status \"fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232\": rpc error: code = NotFound desc = could not find container \"fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232\": container with ID starting with fd7f8a7a076ce649816d472a6c7b1d760fe5e58d1043d288d8cdb9c050eee232 not found: ID does not exist"
Jul 23 22:39:20.332043 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.332022    2579 scope.go:117] "RemoveContainer" containerID="997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184"
Jul 23 22:39:20.332273 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:39:20.332254    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184\": container with ID starting with 997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184 not found: ID does not exist" containerID="997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184"
Jul 23 22:39:20.332315 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.332282    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184"} err="failed to get container status \"997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184\": rpc error: code = NotFound desc = could not find container \"997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184\": container with ID starting with 997e8a37c8acc56e63acb28ee60bf274a196d95c66844ab84bcc8f92046fc184 not found: ID does not exist"
Jul 23 22:39:20.335723 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.335698    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"]
Jul 23 22:39:20.340804 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:20.340768    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-a1850d-openshift-default-5496cc799-jf7c2"]
Jul 23 22:39:21.111614 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:21.111578    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00a473ba-7800-4e53-baca-43db45064258" path="/var/lib/kubelet/pods/00a473ba-7800-4e53-baca-43db45064258/volumes"
Jul 23 22:39:25.639693 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:25.639653    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"]
Jul 23 22:39:25.640283 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:25.640008    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs" podUID="3f414145-66e2-4979-a331-99ebf804f9a1" containerName="maas-api" containerID="cri-o://1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698" gracePeriod=30
Jul 23 22:39:25.661333 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:25.661276    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:26.082705 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.082668    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:39:26.122544 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.122496    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f414145-66e2-4979-a331-99ebf804f9a1-maas-api-tls\") pod \"3f414145-66e2-4979-a331-99ebf804f9a1\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") "
Jul 23 22:39:26.122746 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.122578    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f414145-66e2-4979-a331-99ebf804f9a1-openshift-service-ca\") pod \"3f414145-66e2-4979-a331-99ebf804f9a1\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") "
Jul 23 22:39:26.122746 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.122612    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xpsc\" (UniqueName: \"kubernetes.io/projected/3f414145-66e2-4979-a331-99ebf804f9a1-kube-api-access-2xpsc\") pod \"3f414145-66e2-4979-a331-99ebf804f9a1\" (UID: \"3f414145-66e2-4979-a331-99ebf804f9a1\") "
Jul 23 22:39:26.123050 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.123020    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f414145-66e2-4979-a331-99ebf804f9a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f414145-66e2-4979-a331-99ebf804f9a1" (UID: "3f414145-66e2-4979-a331-99ebf804f9a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 22:39:26.124908 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.124851    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f414145-66e2-4979-a331-99ebf804f9a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f414145-66e2-4979-a331-99ebf804f9a1" (UID: "3f414145-66e2-4979-a331-99ebf804f9a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 22:39:26.125026 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.124882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f414145-66e2-4979-a331-99ebf804f9a1-kube-api-access-2xpsc" (OuterVolumeSpecName: "kube-api-access-2xpsc") pod "3f414145-66e2-4979-a331-99ebf804f9a1" (UID: "3f414145-66e2-4979-a331-99ebf804f9a1"). InnerVolumeSpecName "kube-api-access-2xpsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 22:39:26.224142 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.224104    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f414145-66e2-4979-a331-99ebf804f9a1-maas-api-tls\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:26.224142 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.224136    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f414145-66e2-4979-a331-99ebf804f9a1-openshift-service-ca\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:26.224142 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.224147    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xpsc\" (UniqueName: \"kubernetes.io/projected/3f414145-66e2-4979-a331-99ebf804f9a1-kube-api-access-2xpsc\") on node \"ip-10-0-135-91.ec2.internal\" DevicePath \"\""
Jul 23 22:39:26.344383 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.344350    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f414145-66e2-4979-a331-99ebf804f9a1" containerID="1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698" exitCode=0
Jul 23 22:39:26.344561 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.344398    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"
Jul 23 22:39:26.344561 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.344440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs" event={"ID":"3f414145-66e2-4979-a331-99ebf804f9a1","Type":"ContainerDied","Data":"1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698"}
Jul 23 22:39:26.344561 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.344495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs" event={"ID":"3f414145-66e2-4979-a331-99ebf804f9a1","Type":"ContainerDied","Data":"1c9aaa7de4f5ea278d485143d1e5ab8ca9b93337d230e1f499e44ffbc8b4fe04"}
Jul 23 22:39:26.344561 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.344521    2579 scope.go:117] "RemoveContainer" containerID="1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698"
Jul 23 22:39:26.353118 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.353100    2579 scope.go:117] "RemoveContainer" containerID="1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698"
Jul 23 22:39:26.353406 ip-10-0-135-91 kubenswrapper[2579]: E0723 22:39:26.353386    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698\": container with ID starting with 1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698 not found: ID does not exist" containerID="1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698"
Jul 23 22:39:26.353454 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.353416    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698"} err="failed to get container status \"1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698\": rpc error: code = NotFound desc = could not find container \"1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698\": container with ID starting with 1fa43f671b5bf11de5da7268b002921a0402adfff8d5e4d67596ff726f927698 not found: ID does not exist"
Jul 23 22:39:26.366045 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.366015    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"]
Jul 23 22:39:26.370111 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:26.370086    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-091ba5-8f8687544-rvfrs"]
Jul 23 22:39:27.111602 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:27.111555    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f414145-66e2-4979-a331-99ebf804f9a1" path="/var/lib/kubelet/pods/3f414145-66e2-4979-a331-99ebf804f9a1/volumes"
Jul 23 22:39:30.237324 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:30.237264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:30.237324 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:30.237283    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"]
Jul 23 22:39:30.239334 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:30.239313    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-091ba5-k99qt"]
Jul 23 22:39:30.626110 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:30.626028    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:31.112039 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:31.112007    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfb33a4c-b972-4818-a151-d50b9eb55b76" path="/var/lib/kubelet/pods/bfb33a4c-b972-4818-a151-d50b9eb55b76/volumes"
Jul 23 22:39:36.687612 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:36.687521    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:36.757348 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:36.757295    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:36.988733 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:36.988653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:37.222146 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:37.222108    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:44.095661 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:44.095626    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6864c64679-4cjcz_bc156a06-fd78-4756-9283-569e236dcc64/copy-manifests/0.log"
Jul 23 22:39:44.108692 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:44.108662    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6864c64679-4cjcz_bc156a06-fd78-4756-9283-569e236dcc64/manager/0.log"
Jul 23 22:39:44.615607 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:44.615573    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76bbd449ff-bf66q_7886aac9-112c-4ad6-ae02-d7cff7963ce8/maas-api/0.log"
Jul 23 22:39:44.729728 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:44.729696    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747415-8r5jn_d18c613f-f81d-4528-a09b-8036ddaa8b1c/cleanup/0.log"
Jul 23 22:39:44.957514 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:44.957459    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-fhnpl_c1bdfa83-6de1-43fa-9432-8da3566bbc73/postgres/0.log"
Jul 23 22:39:45.474076 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:45.474044    2579 scope.go:117] "RemoveContainer" containerID="0758c8d945b07d346c9ef6b4f47cb088318f2fcab09aaff8e485a454972bdbf2"
Jul 23 22:39:47.940734 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:47.940703    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b78d9f867-wc99x_0bcc10ec-7c22-44dc-b85a-00b4e3adf48e/kube-auth-proxy/0.log"
Jul 23 22:39:48.323135 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:48.323096    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-6pshb_f667c912-b23d-4b18-be7b-6f4574133175/payload-processing/0.log"
Jul 23 22:39:48.943698 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:48.943666    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-tnkf4_0a423e18-04e1-4b2f-9a2e-87653ce9214a/main/0.log"
Jul 23 22:39:49.416852 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:49.416821    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5mvbs_029169b6-1d2a-44cd-b9a9-a8d328300040/main/0.log"
Jul 23 22:39:55.026664 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.026631    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fvbz2/must-gather-p78x6"]
Jul 23 22:39:55.027066 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.026930    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy"
Jul 23 22:39:55.027066 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.026941    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy"
Jul 23 22:39:55.027066 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.026950    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfb33a4c-b972-4818-a151-d50b9eb55b76" containerName="revoke-keys"
Jul 23 22:39:55.027066 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.026958    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfb33a4c-b972-4818-a151-d50b9eb55b76" containerName="revoke-keys"
Jul 23 22:39:55.027066 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.026966    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f414145-66e2-4979-a331-99ebf804f9a1" containerName="maas-api"
Jul 23 22:39:55.027066 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.026973    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f414145-66e2-4979-a331-99ebf804f9a1" containerName="maas-api"
Jul 23 22:39:55.027066 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.027030    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f414145-66e2-4979-a331-99ebf804f9a1" containerName="maas-api"
Jul 23 22:39:55.027066 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.027036    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfb33a4c-b972-4818-a151-d50b9eb55b76" containerName="revoke-keys"
Jul 23 22:39:55.027066 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.027043    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy"
Jul 23 22:39:55.027066 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.027050    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy"
Jul 23 22:39:55.027397 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.027101    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy"
Jul 23 22:39:55.027397 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.027107    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a473ba-7800-4e53-baca-43db45064258" containerName="istio-proxy"
Jul 23 22:39:55.030070 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.030053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvbz2/must-gather-p78x6"
Jul 23 22:39:55.032488 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.032455    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fvbz2\"/\"openshift-service-ca.crt\""
Jul 23 22:39:55.032488 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.032455    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fvbz2\"/\"default-dockercfg-fnxfx\""
Jul 23 22:39:55.033171 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.033152    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fvbz2\"/\"kube-root-ca.crt\""
Jul 23 22:39:55.040423 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.040373    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvbz2/must-gather-p78x6"]
Jul 23 22:39:55.041731 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.041695    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:55.136740 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.136691    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log" limit=1024
Jul 23 22:39:55.173536 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.173502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzn5\" (UniqueName: \"kubernetes.io/projected/ba84fb57-5ff1-4c4d-af33-3a172425b015-kube-api-access-gfzn5\") pod \"must-gather-p78x6\" (UID: \"ba84fb57-5ff1-4c4d-af33-3a172425b015\") " pod="openshift-must-gather-fvbz2/must-gather-p78x6"
Jul 23 22:39:55.173663 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.173611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ba84fb57-5ff1-4c4d-af33-3a172425b015-must-gather-output\") pod \"must-gather-p78x6\" (UID: \"ba84fb57-5ff1-4c4d-af33-3a172425b015\") " pod="openshift-must-gather-fvbz2/must-gather-p78x6"
Jul 23 22:39:55.274614 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.274572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ba84fb57-5ff1-4c4d-af33-3a172425b015-must-gather-output\") pod \"must-gather-p78x6\" (UID: \"ba84fb57-5ff1-4c4d-af33-3a172425b015\") " pod="openshift-must-gather-fvbz2/must-gather-p78x6"
Jul 23 22:39:55.274800 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.274651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzn5\" (UniqueName: \"kubernetes.io/projected/ba84fb57-5ff1-4c4d-af33-3a172425b015-kube-api-access-gfzn5\") pod \"must-gather-p78x6\" (UID: \"ba84fb57-5ff1-4c4d-af33-3a172425b015\") " pod="openshift-must-gather-fvbz2/must-gather-p78x6"
Jul 23 22:39:55.275059 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.275036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ba84fb57-5ff1-4c4d-af33-3a172425b015-must-gather-output\") pod \"must-gather-p78x6\" (UID: \"ba84fb57-5ff1-4c4d-af33-3a172425b015\") " pod="openshift-must-gather-fvbz2/must-gather-p78x6"
Jul 23 22:39:55.283222 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.283151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzn5\" (UniqueName: \"kubernetes.io/projected/ba84fb57-5ff1-4c4d-af33-3a172425b015-kube-api-access-gfzn5\") pod \"must-gather-p78x6\" (UID: \"ba84fb57-5ff1-4c4d-af33-3a172425b015\") " pod="openshift-must-gather-fvbz2/must-gather-p78x6"
Jul 23 22:39:55.339332 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.339295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvbz2/must-gather-p78x6"
Jul 23 22:39:55.476939 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:55.476760    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvbz2/must-gather-p78x6"]
Jul 23 22:39:55.479643 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:39:55.479612    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba84fb57_5ff1_4c4d_af33_3a172425b015.slice/crio-60db2c20b36925fbf87c720837a22ff1940708a62354bad5efec9e91e22d766d WatchSource:0}: Error finding container 60db2c20b36925fbf87c720837a22ff1940708a62354bad5efec9e91e22d766d: Status 404 returned error can't find the container with id 60db2c20b36925fbf87c720837a22ff1940708a62354bad5efec9e91e22d766d
Jul 23 22:39:56.475032 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:56.475002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvbz2/must-gather-p78x6" event={"ID":"ba84fb57-5ff1-4c4d-af33-3a172425b015","Type":"ContainerStarted","Data":"bb8e4e562ee0329b80761fb8e921a0a212426bae47b6b55043ca62017964d33a"}
Jul 23 22:39:56.475032 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:56.475054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvbz2/must-gather-p78x6" event={"ID":"ba84fb57-5ff1-4c4d-af33-3a172425b015","Type":"ContainerStarted","Data":"60db2c20b36925fbf87c720837a22ff1940708a62354bad5efec9e91e22d766d"}
Jul 23 22:39:57.483594 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:57.483550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvbz2/must-gather-p78x6" event={"ID":"ba84fb57-5ff1-4c4d-af33-3a172425b015","Type":"ContainerStarted","Data":"e0a53e5e9b9a1cbb72f3866cab3ed4b47015780c13259523cdef7a3ae16cb6ed"}
Jul 23 22:39:57.499899 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:57.499335    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fvbz2/must-gather-p78x6" podStartSLOduration=1.59975952 podStartE2EDuration="2.499312685s" totalImagesPullingTime="899.553165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 22:39:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:39:57.498177749 +0000 UTC m=+2892.888570365" watchObservedRunningTime="2026-07-23 22:39:57.499312685 +0000 UTC m=+2892.889705302"
Jul 23 22:39:58.045354 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:58.045324    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-455s5_e38142be-62e8-49b7-af8d-cad1fcb9b7c5/global-pull-secret-syncer/0.log"
Jul 23 22:39:58.135945 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:58.135911    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m97rx_f9eb7954-e1c1-4140-a634-51cb814450cd/konnectivity-agent/0.log"
Jul 23 22:39:58.230436 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:39:58.230404    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-91.ec2.internal_599b3085f9eb491ede314f11a46d9b2b/haproxy/0.log"
Jul 23 22:40:05.656569 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:05.656467    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v7wch_a29f1bfa-eccf-4f32-8209-44086ce2bf58/node-exporter/0.log"
Jul 23 22:40:05.689141 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:05.689105    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v7wch_a29f1bfa-eccf-4f32-8209-44086ce2bf58/kube-rbac-proxy/0.log"
Jul 23 22:40:05.715917 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:05.715889    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v7wch_a29f1bfa-eccf-4f32-8209-44086ce2bf58/init-textfile/0.log"
Jul 23 22:40:05.848694 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:05.848663    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3222819f-a6a5-42dd-b48f-adb8bd198885/prometheus/0.log"
Jul 23 22:40:05.874636 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:05.874580    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3222819f-a6a5-42dd-b48f-adb8bd198885/config-reloader/0.log"
Jul 23 22:40:05.898756 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:05.898730    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3222819f-a6a5-42dd-b48f-adb8bd198885/thanos-sidecar/0.log"
Jul 23 22:40:05.928473 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:05.928394    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3222819f-a6a5-42dd-b48f-adb8bd198885/kube-rbac-proxy-web/0.log"
Jul 23 22:40:05.951306 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:05.951279    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3222819f-a6a5-42dd-b48f-adb8bd198885/kube-rbac-proxy/0.log"
Jul 23 22:40:05.973979 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:05.973953    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3222819f-a6a5-42dd-b48f-adb8bd198885/kube-rbac-proxy-thanos/0.log"
Jul 23 22:40:06.001566 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:06.001534    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3222819f-a6a5-42dd-b48f-adb8bd198885/init-config-reloader/0.log"
Jul 23 22:40:08.149803 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.149742    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"]
Jul 23 22:40:08.155883 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.155836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.164581 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.164278    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"]
Jul 23 22:40:08.204068 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.204019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-podres\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.204481 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.204455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-sys\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.205363 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.205342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-proc\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.205532 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.205516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-lib-modules\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.205701 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.205682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmxfp\" (UniqueName: \"kubernetes.io/projected/e30f30fe-9469-4397-b130-5185c7046573-kube-api-access-vmxfp\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.307372 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.307323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-podres\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.307575 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.307433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-sys\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.307575 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.307494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-proc\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.307575 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.307516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-lib-modules\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.307575 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.307540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmxfp\" (UniqueName: \"kubernetes.io/projected/e30f30fe-9469-4397-b130-5185c7046573-kube-api-access-vmxfp\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.308093 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.308054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-podres\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.308211 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.308140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-sys\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.308211 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.308188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-proc\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.308315 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.308271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30f30fe-9469-4397-b130-5185c7046573-lib-modules\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.316833 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.316784    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmxfp\" (UniqueName: \"kubernetes.io/projected/e30f30fe-9469-4397-b130-5185c7046573-kube-api-access-vmxfp\") pod \"perf-node-gather-daemonset-nlvnc\" (UID: \"e30f30fe-9469-4397-b130-5185c7046573\") " pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.470421 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.470386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:08.654547 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:08.654494    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"]
Jul 23 22:40:08.659022 ip-10-0-135-91 kubenswrapper[2579]: W0723 22:40:08.658989    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode30f30fe_9469_4397_b130_5185c7046573.slice/crio-bbcbeeafeb7d5178a468b07c0463ab15c21e59878976bf5e10e96a040f6f7140 WatchSource:0}: Error finding container bbcbeeafeb7d5178a468b07c0463ab15c21e59878976bf5e10e96a040f6f7140: Status 404 returned error can't find the container with id bbcbeeafeb7d5178a468b07c0463ab15c21e59878976bf5e10e96a040f6f7140
Jul 23 22:40:09.558573 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:09.558531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc" event={"ID":"e30f30fe-9469-4397-b130-5185c7046573","Type":"ContainerStarted","Data":"5c81249e507dc86a92d70f3b83c18c0ba652d0df28a7bd068b1159c142afac2f"}
Jul 23 22:40:09.559088 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:09.558584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc" event={"ID":"e30f30fe-9469-4397-b130-5185c7046573","Type":"ContainerStarted","Data":"bbcbeeafeb7d5178a468b07c0463ab15c21e59878976bf5e10e96a040f6f7140"}
Jul 23 22:40:09.559088 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:09.558615    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:09.575702 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:09.575644    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc" podStartSLOduration=1.575625446 podStartE2EDuration="1.575625446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 22:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 22:40:09.575468312 +0000 UTC m=+2904.965860928" watchObservedRunningTime="2026-07-23 22:40:09.575625446 +0000 UTC m=+2904.966018120"
Jul 23 22:40:10.794475 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:10.794448    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l5bhg_59368801-3baf-4007-bf31-62e00a4a8306/dns/0.log"
Jul 23 22:40:10.814676 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:10.814650    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l5bhg_59368801-3baf-4007-bf31-62e00a4a8306/kube-rbac-proxy/0.log"
Jul 23 22:40:10.837709 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:10.837679    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j8z9l_53fb3d87-a41e-448b-b0f5-264a123a6811/dns-node-resolver/0.log"
Jul 23 22:40:11.492006 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:11.491969    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jtp7k_ba440199-82c0-4d4a-a4ef-29dc6ce4720a/node-ca/0.log"
Jul 23 22:40:12.666161 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:12.666134    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b78d9f867-wc99x_0bcc10ec-7c22-44dc-b85a-00b4e3adf48e/kube-auth-proxy/0.log"
Jul 23 22:40:12.798406 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:12.798376    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-6pshb_f667c912-b23d-4b18-be7b-6f4574133175/payload-processing/0.log"
Jul 23 22:40:13.557396 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:13.557366    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rpp2m_b3fb3e9c-a9ef-4631-b30b-645d9e5e4388/serve-healthcheck-canary/0.log"
Jul 23 22:40:14.141246 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:14.141208    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jh4m_2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5/kube-rbac-proxy/0.log"
Jul 23 22:40:14.164776 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:14.164738    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jh4m_2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5/exporter/0.log"
Jul 23 22:40:14.200408 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:14.199381    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5jh4m_2db9d3b8-fd1b-42d9-96ee-9eae80cfdbe5/extractor/0.log"
Jul 23 22:40:15.575420 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:15.575390    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fvbz2/perf-node-gather-daemonset-nlvnc"
Jul 23 22:40:17.134992 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:17.134843    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6864c64679-4cjcz_bc156a06-fd78-4756-9283-569e236dcc64/manager/0.log"
Jul 23 22:40:17.154427 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:17.154400    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6864c64679-4cjcz_bc156a06-fd78-4756-9283-569e236dcc64/copy-manifests/0.log"
Jul 23 22:40:18.634057 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:18.634026    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7575c4d6b8-792bd_81b8750e-08ea-4ae7-ac3b-effa41bc7fe4/manager/0.log"
Jul 23 22:40:19.201033 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:19.201003    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b78d9f867-wc99x_0bcc10ec-7c22-44dc-b85a-00b4e3adf48e/kube-auth-proxy/0.log"
Jul 23 22:40:19.287580 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:19.287540    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-6pshb_f667c912-b23d-4b18-be7b-6f4574133175/payload-processing/0.log"
Jul 23 22:40:20.667875 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:20.667800    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b78d9f867-wc99x_0bcc10ec-7c22-44dc-b85a-00b4e3adf48e/kube-auth-proxy/0.log"
Jul 23 22:40:20.761946 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:20.761890    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-6pshb_f667c912-b23d-4b18-be7b-6f4574133175/payload-processing/0.log"
Jul 23 22:40:25.233378 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:25.233347    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khkxf_6eca0599-f929-4362-aedc-edf38f96e1d1/kube-multus-additional-cni-plugins/0.log"
Jul 23 22:40:25.254816 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:25.254787    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khkxf_6eca0599-f929-4362-aedc-edf38f96e1d1/egress-router-binary-copy/0.log"
Jul 23 22:40:25.275371 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:25.275341    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khkxf_6eca0599-f929-4362-aedc-edf38f96e1d1/cni-plugins/0.log"
Jul 23 22:40:25.298089 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:25.298062    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khkxf_6eca0599-f929-4362-aedc-edf38f96e1d1/bond-cni-plugin/0.log"
Jul 23 22:40:25.317498 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:25.317467    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khkxf_6eca0599-f929-4362-aedc-edf38f96e1d1/routeoverride-cni/0.log"
Jul 23 22:40:25.341316 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:25.341280    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khkxf_6eca0599-f929-4362-aedc-edf38f96e1d1/whereabouts-cni-bincopy/0.log"
Jul 23 22:40:25.362824 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:25.362796    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-khkxf_6eca0599-f929-4362-aedc-edf38f96e1d1/whereabouts-cni/0.log"
Jul 23 22:40:25.749521 ip-10-0-135-91 kubenswrapper[2579]: I0723 22:40:25.749465    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n9jpn_5f47c3f9-ee00-43f7-b343-d26d37c20631/kube-multus/0.log"